Building

Started building at 2021/12/14 10:37:44
Calculating base
Updating mirror
Basing run on 7.1.0-1905 31e79f513
Updating tree for run 14.12.2021-10.37
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 8cc3b2c, changes since last good build: 
 8cc3b2cb Merge remote-tracking branch 'couchbase/unstable' into HEAD
 12a78d6d MB-49079 (7.1.0 1861) Rebalance: Omit inconsistent indexes from Planner
kv_engine is at 2dc5ee9, changes since last good build: none
ns_server is at abe4a02, changes since last good build: 
 abe4a0276 MB-50021 populate retry_rebalance key in default config
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 45ee69a, changes since last good build: 
 45ee69a6f MB-49703 add create/drop scope/collection test cases with scope_admin role
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable e1b5fe2a] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable d96e42d0] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 4bb945d9] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 11310ce9] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 19422b83] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 043a08c6] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 33ce867f] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 83a3442a] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 9e038f1] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/14 10:54:27
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/14 10:55:59 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/14 10:56:09 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 10:56:14 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/14 10:56:14 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 10:56:20 Removing node(s): [] from the cluster
2021/12/14 10:56:25 Rebalance progress: 0
2021/12/14 10:56:30 Rebalance progress: 0.0904224537037037
2021/12/14 10:56:35 Rebalance progress: 4.756221064814815
2021/12/14 10:56:40 Rebalance progress: 9.25925925925926
2021/12/14 10:56:45 Rebalance progress: 12.91232638888889
2021/12/14 10:56:50 Rebalance progress: 17.48770254629629
2021/12/14 10:56:55 Rebalance progress: 20.83333333333333
2021/12/14 10:57:00 Rebalance progress: 25.46296296296296
2021/12/14 10:57:05 Rebalance progress: 100
2021/12/14 10:57:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:57:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:57:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:57:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:58:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:58:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:58:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:58:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:59:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:59:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:59:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 10:59:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:00:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:00:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:00:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:00:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:01:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:01:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:01:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/14 11:01:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/14 11:02:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:02:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:02:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:02:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:03:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:03:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:03:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:03:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:04:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:04:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:04:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:04:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:05:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:05:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:05:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:05:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/14 11:06:10 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:06:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:06:16 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 11:06:21 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 11:06:26 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 11:06:31 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/14 11:06:31 App: TestAdvancedGetOps got deployed
2021/12/14 11:06:31 Starting bucket ops to source bucket
2021/12/14 11:06:36 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:06:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:06:47 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 11:06:47 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:06:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:06:47 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:06:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:06:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 11:06:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 11:07:02 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 11:07:07 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/14 11:07:07 App: TestAdvancedGetOps got un-deployed
2021/12/14 11:07:07 No eventing-con process running
2021/12/14 11:07:07 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/14 11:07:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:07:18 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/14 11:07:23 Waiting for dst bucket item count to get to: 0 curr count: 28
2021/12/14 11:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:07:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:07:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (325.13s)
=== RUN   TestAdvancedInsertOps
2021/12/14 11:07:35 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:07:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:07:41 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:07:46 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:07:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:07:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:07:56 App: TestAdvancedInsertOps got deployed
2021/12/14 11:07:56 Starting bucket ops to source bucket
2021/12/14 11:08:02 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 11:08:07 Waiting for dst bucket item count to get to: 1024 curr count: 1001
2021/12/14 11:08:12 Waiting for dst bucket item count to get to: 1024 curr count: 1001
2021/12/14 11:08:12 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 11:08:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:08:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:08:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:08:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:08:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:08:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:08:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:08:32 App: TestAdvancedInsertOps got un-deployed
2021/12/14 11:08:32 No eventing-con process running
2021/12/14 11:08:32 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 11:08:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:08:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:08:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 11:08:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:08:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:09:01 Waiting for dst bucket item count to get to: 0 curr count: 976
2021/12/14 11:09:06 Waiting for dst bucket item count to get to: 0 curr count: 976
2021/12/14 11:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:09:06 Testing insert operation on source bucket
2021/12/14 11:09:06 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:09:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:09:12 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:09:17 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:09:22 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:09:27 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/14 11:09:27 App: TestAdvancedInsertOps got deployed
2021/12/14 11:09:27 Starting bucket ops to source bucket
2021/12/14 11:09:33 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 11:09:38 Waiting for dst bucket item count to get to: 2048 curr count: 1045
2021/12/14 11:09:43 Waiting for dst bucket item count to get to: 2048 curr count: 1045
2021/12/14 11:09:43 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 11:09:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:09:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:09:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:09:48 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:09:53 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:09:58 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:10:03 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/14 11:10:03 App: TestAdvancedInsertOps got un-deployed
2021/12/14 11:10:03 No eventing-con process running
2021/12/14 11:10:03 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/14 11:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:10:15 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 11:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (166.53s)
=== RUN   TestAdvancedUpsertOps
2021/12/14 11:10:22 Starting bucket ops to source bucket
2021/12/14 11:10:22 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:10:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:10:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:10:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:10:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:10:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:10:44 App: TestAdvancedUpsertOps got deployed
2021/12/14 11:10:44 Starting bucket ops to source bucket
2021/12/14 11:10:50 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:10:55 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:10:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:10:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:10:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:11:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:11:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:11:10 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:11:15 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:11:15 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 11:11:15 No eventing-con process running
2021/12/14 11:11:15 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 11:11:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:11:26 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:11:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:11:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:11:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:11:33 Testing upsert operation on source bucket
2021/12/14 11:11:33 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:11:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:11:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:11:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:11:50 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:11:55 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/14 11:11:55 App: TestAdvancedUpsertOps got deployed
2021/12/14 11:11:55 Starting bucket ops to source bucket
2021/12/14 11:11:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:11:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:11:56 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:11:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:12:01 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:12:06 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:12:11 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:12:16 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/14 11:12:16 App: TestAdvancedUpsertOps got un-deployed
2021/12/14 11:12:16 No eventing-con process running
2021/12/14 11:12:16 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/14 11:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:12:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:12:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.01s)
=== RUN   TestAdvancedReplaceOps
2021/12/14 11:12:30 Starting bucket ops to source bucket
2021/12/14 11:12:31 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:12:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:12:37 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:12:42 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:12:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:12:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:12:52 App: TestAdvancedReplaceOps got deployed
2021/12/14 11:12:52 Starting bucket ops to source bucket
2021/12/14 11:12:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:13:03 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:13:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:13:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:13:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:13:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:13:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:13:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:13:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:13:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:13:23 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 11:13:23 No eventing-con process running
2021/12/14 11:13:23 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 11:13:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:13:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:13:42 Testing upsert operation on source bucket
2021/12/14 11:13:42 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:13:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:13:48 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:13:53 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:13:58 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:14:03 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/14 11:14:03 App: TestAdvancedReplaceOps got deployed
2021/12/14 11:14:03 Starting bucket ops to source bucket
2021/12/14 11:14:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:14:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:14:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:14:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:14:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:14:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:14:19 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:14:24 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/14 11:14:24 App: TestAdvancedReplaceOps got un-deployed
2021/12/14 11:14:24 No eventing-con process running
2021/12/14 11:14:24 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/14 11:14:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:14:36 Waiting for dst bucket item count to get to: 0 curr count: 394
2021/12/14 11:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:14:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:14:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.92s)
=== RUN   TestAdvancedDeleteOps
2021/12/14 11:14:43 Starting bucket ops to source bucket
2021/12/14 11:14:43 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:14:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:14:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:14:54 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:14:59 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:15:04 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:15:04 App: TestAdvancedDeleteOps got deployed
2021/12/14 11:15:04 Starting bucket ops to source bucket
2021/12/14 11:15:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:15:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:15:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:15:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:15:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:15:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:15:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:15:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:15:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:15:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:15:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 11:15:34 No eventing-con process running
2021/12/14 11:15:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 11:15:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:15:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:15:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:15:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:15:53 Testing delete operation on source bucket
2021/12/14 11:15:53 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:15:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/14 11:15:59 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:16:04 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:16:09 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:16:14 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/14 11:16:14 App: TestAdvancedDeleteOps got deployed
2021/12/14 11:16:14 Starting bucket ops to source bucket
2021/12/14 11:16:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:16:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:16:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:16:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:16:19 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:16:24 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:16:29 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:16:34 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/14 11:16:34 App: TestAdvancedDeleteOps got un-deployed
2021/12/14 11:16:34 No eventing-con process running
2021/12/14 11:16:34 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/14 11:16:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:16:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:16:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.83s)
=== RUN   TestEnoentAdvancedGet
2021/12/14 11:16:48 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:16:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 11:16:55 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:17:00 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:17:05 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:17:10 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:17:10 App: TestEnoentAdvancedGet got deployed
2021/12/14 11:17:10 Starting bucket ops to source bucket
2021/12/14 11:17:15 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:17:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:17:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 11:17:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:17:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:17:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:17:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:17:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:17:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:17:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:17:45 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:17:45 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 11:17:45 No eventing-con process running
2021/12/14 11:17:45 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 11:17:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:17:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:17:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:17:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:18:08 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/14 11:18:14 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/14 11:18:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:18:14 Testing get enoent operation on source bucket
2021/12/14 11:18:14 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:18:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/14 11:18:20 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:18:25 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:18:30 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:18:35 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/14 11:18:35 App: TestEnoentAdvancedGet got deployed
2021/12/14 11:18:35 Starting bucket ops to source bucket
2021/12/14 11:18:40 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 11:18:45 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 11:18:45 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 11:18:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:18:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:18:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:18:50 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:18:55 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:19:00 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:19:05 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/14 11:19:05 App: TestEnoentAdvancedGet got un-deployed
2021/12/14 11:19:05 No eventing-con process running
2021/12/14 11:19:05 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/14 11:19:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:19:17 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 11:19:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:19:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:19:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.05s)
=== RUN   TestEnoentAdvancedReplace
2021/12/14 11:19:24 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:19:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:19:30 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:19:35 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:19:40 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:19:45 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:19:45 App: TestEnoentAdvancedReplace got deployed
2021/12/14 11:19:45 Starting bucket ops to source bucket
2021/12/14 11:19:50 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:19:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 11:19:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:19:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:19:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:19:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:20:00 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:20:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:20:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:20:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:20:15 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 11:20:15 No eventing-con process running
2021/12/14 11:20:15 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 11:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:20:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:20:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:20:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:20:38 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/14 11:20:43 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/14 11:20:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:20:43 Testing get enoent operation on source bucket
2021/12/14 11:20:43 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:20:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:20:50 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:20:55 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:21:00 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:21:05 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/14 11:21:05 App: TestEnoentAdvancedReplace got deployed
2021/12/14 11:21:05 Starting bucket ops to source bucket
2021/12/14 11:21:10 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 11:21:15 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 11:21:15 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 11:21:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:21:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:21:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:21:20 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:21:25 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:21:30 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:21:35 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/14 11:21:35 App: TestEnoentAdvancedReplace got un-deployed
2021/12/14 11:21:35 No eventing-con process running
2021/12/14 11:21:35 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/14 11:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:21:47 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 11:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:21:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:21:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.03s)
=== RUN   TestEnoentAdvancedDelete
2021/12/14 11:21:54 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:21:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 11:22:00 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:22:05 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:22:10 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:22:15 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:22:15 App: TestEnoentAdvancedDelete got deployed
2021/12/14 11:22:15 Starting bucket ops to source bucket
2021/12/14 11:22:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:22:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 11:22:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:22:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:22:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:22:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:22:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:22:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:22:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:22:45 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:22:45 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 11:22:45 No eventing-con process running
2021/12/14 11:22:45 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 11:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:22:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:22:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:23:08 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/14 11:23:13 Waiting for dst bucket item count to get to: 0 curr count: 33
2021/12/14 11:23:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:23:13 Testing get enoent operation on source bucket
2021/12/14 11:23:13 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:23:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 11:23:20 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:23:25 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:23:30 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:23:35 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/14 11:23:35 App: TestEnoentAdvancedDelete got deployed
2021/12/14 11:23:35 Starting bucket ops to source bucket
2021/12/14 11:23:40 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/14 11:23:45 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/14 11:23:45 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/14 11:23:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:23:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:23:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:23:50 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:23:55 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:24:00 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:24:05 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/14 11:24:05 App: TestEnoentAdvancedDelete got un-deployed
2021/12/14 11:24:05 No eventing-con process running
2021/12/14 11:24:05 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/14 11:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:24:17 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 11:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:24:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:24:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.96s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/14 11:24:23 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:24:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 11:24:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:24:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:24:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:24:45 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:24:45 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 11:24:45 Starting bucket ops to source bucket
2021/12/14 11:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:24:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:24:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:24:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:24:50 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:24:55 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:25:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:25:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:25:05 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 11:25:05 No eventing-con process running
2021/12/14 11:25:05 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 11:25:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:25:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:25:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:25:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:25:19 Pumping document in the destination bucket
2021/12/14 11:25:19 Starting bucket ops to source bucket
2021/12/14 11:25:19 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:25:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/14 11:25:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:25:30 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:25:35 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:25:40 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/14 11:25:40 App: TestAdvancedInsertKeyExist got deployed
2021/12/14 11:25:40 Starting bucket ops to source bucket
2021/12/14 11:25:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:25:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:25:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:25:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:25:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:25:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:25:56 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:26:01 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:26:06 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:26:11 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/14 11:26:11 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/14 11:26:11 No eventing-con process running
2021/12/14 11:26:11 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/14 11:26:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:26:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:26:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:26:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:26:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.65s)
=== RUN   TestExipryGet
2021/12/14 11:26:34 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:26:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 11:26:40 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:26:45 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:26:50 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:26:55 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:26:55 App: TestExipryGet got deployed
2021/12/14 11:26:55 Starting bucket ops to source bucket
2021/12/14 11:27:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:27:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:27:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:27:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:27:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:27:46 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:27:51 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:27:56 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:28:01 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:28:01 App: TestExipryGet got un-deployed
2021/12/14 11:28:01 No eventing-con process running
2021/12/14 11:28:01 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 11:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:28:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:28:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:28:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:28:15 Pumping document in the destination bucket for expiry get
2021/12/14 11:28:15 Starting bucket ops to source bucket
2021/12/14 11:28:15 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:28:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/14 11:28:21 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:28:26 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:28:31 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:28:36 Waiting for app: TestExipryGet to get deployed
2021/12/14 11:28:36 App: TestExipryGet got deployed
2021/12/14 11:28:36 Starting bucket ops to source bucket
2021/12/14 11:29:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:29:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:29:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:29:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:29:27 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:29:32 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:29:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:29:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/14 11:29:42 App: TestExipryGet got un-deployed
2021/12/14 11:29:42 No eventing-con process running
2021/12/14 11:29:42 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/14 11:29:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:29:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:29:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:29:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:30:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:30:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (211.02s)
=== RUN   TestExipryInsert
2021/12/14 11:30:05 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:30:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 11:30:11 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:30:16 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:30:21 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:30:26 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:30:26 App: TestExipryInsert got deployed
2021/12/14 11:30:26 Starting bucket ops to source bucket
2021/12/14 11:31:12 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/14 11:31:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:31:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:31:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:31:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:31:17 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:31:22 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:31:27 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:31:32 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:31:32 App: TestExipryInsert got un-deployed
2021/12/14 11:31:32 No eventing-con process running
2021/12/14 11:31:32 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 11:31:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:31:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:31:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:31:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:31:46 Testing on the destination bucket for expiry insert
2021/12/14 11:31:46 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:31:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/14 11:31:52 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:31:57 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:32:02 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:32:07 Waiting for app: TestExipryInsert to get deployed
2021/12/14 11:32:07 App: TestExipryInsert got deployed
2021/12/14 11:32:07 Starting bucket ops to source bucket
2021/12/14 11:32:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:32:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:32:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:32:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:32:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:32:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:33:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:33:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:33:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/14 11:33:12 App: TestExipryInsert got un-deployed
2021/12/14 11:33:12 No eventing-con process running
2021/12/14 11:33:12 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/14 11:33:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:33:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:33:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:33:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:33:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:33:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.71s)
=== RUN   TestExipryReplace
2021/12/14 11:33:36 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:33:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:33:42 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:33:47 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:33:52 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:33:57 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:33:57 App: TestExipryReplace got deployed
2021/12/14 11:33:57 Starting bucket ops to source bucket
2021/12/14 11:34:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:34:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:34:52 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:34:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:34:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:34:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:34:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:34:57 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:35:02 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:35:07 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:35:12 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:35:12 App: TestExipryReplace got un-deployed
2021/12/14 11:35:12 No eventing-con process running
2021/12/14 11:35:12 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 11:35:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:35:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:35:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:35:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:35:27 Testing on the destination bucket for expiry insert
2021/12/14 11:35:27 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:35:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:35:33 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:35:38 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:35:43 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:35:48 Waiting for app: TestExipryReplace to get deployed
2021/12/14 11:35:48 App: TestExipryReplace got deployed
2021/12/14 11:35:48 Starting bucket ops to source bucket
2021/12/14 11:36:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:36:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:36:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:36:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:36:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:36:43 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:36:48 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:36:53 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:36:58 Waiting for app: TestExipryReplace to get un-deployed
2021/12/14 11:36:58 App: TestExipryReplace got un-deployed
2021/12/14 11:36:58 No eventing-con process running
2021/12/14 11:36:58 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/14 11:37:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:37:10 Waiting for dst bucket item count to get to: 0 curr count: 55
2021/12/14 11:37:15 Waiting for dst bucket item count to get to: 0 curr count: 55
2021/12/14 11:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:37:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (225.76s)
=== RUN   TestExipryUpsert
2021/12/14 11:37:22 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:37:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 11:37:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:37:33 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:37:38 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:37:43 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:37:43 App: TestExipryUpsert got deployed
2021/12/14 11:37:43 Starting bucket ops to source bucket
2021/12/14 11:38:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:38:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:38:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:38:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:38:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:38:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:38:38 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:38:43 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:38:48 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:38:53 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:38:53 App: TestExipryUpsert got un-deployed
2021/12/14 11:38:53 No eventing-con process running
2021/12/14 11:38:53 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 11:38:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:39:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:39:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:39:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:39:07 Testing on the destination bucket for expiry insert
2021/12/14 11:39:07 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:39:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/14 11:39:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:39:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:39:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:39:28 Waiting for app: TestExipryUpsert to get deployed
2021/12/14 11:39:28 App: TestExipryUpsert got deployed
2021/12/14 11:39:28 Starting bucket ops to source bucket
2021/12/14 11:40:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:40:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:40:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:40:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:40:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:40:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:40:24 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:40:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:40:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:40:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/14 11:40:39 App: TestExipryUpsert got un-deployed
2021/12/14 11:40:39 No eventing-con process running
2021/12/14 11:40:39 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/14 11:40:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:40:51 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/14 11:40:56 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/14 11:40:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:41:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.69s)
=== RUN   TestCasReplace
2021/12/14 11:41:02 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:41:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:41:09 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:41:14 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:41:19 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:41:24 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:41:24 App: TestCasReplace got deployed
2021/12/14 11:41:24 Starting bucket ops to source bucket
2021/12/14 11:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:41:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:41:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:41:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:41:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:41:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:41:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:41:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:41:44 App: TestCasReplace got un-deployed
2021/12/14 11:41:44 No eventing-con process running
2021/12/14 11:41:44 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 11:41:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:41:56 Waiting for dst bucket item count to get to: 0 curr count: 9
2021/12/14 11:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:42:03 Testing on the destination bucket for upsert with cas
2021/12/14 11:42:03 Starting bucket ops to source bucket
2021/12/14 11:42:03 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:42:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/14 11:42:09 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:42:14 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:42:19 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:42:24 Waiting for app: TestCasReplace to get deployed
2021/12/14 11:42:24 App: TestCasReplace got deployed
2021/12/14 11:42:24 Starting bucket ops to source bucket
2021/12/14 11:42:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:42:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:42:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:42:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:42:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:42:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:42:39 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:42:44 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:42:49 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:42:54 Waiting for app: TestCasReplace to get un-deployed
2021/12/14 11:42:54 App: TestCasReplace got un-deployed
2021/12/14 11:42:54 No eventing-con process running
2021/12/14 11:42:54 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/14 11:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:43:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:43:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:43:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.30s)
=== RUN   TestCasDelete
2021/12/14 11:43:13 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:43:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 11:43:19 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:43:24 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:43:29 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:43:34 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:43:34 App: TestCasDelete got deployed
2021/12/14 11:43:34 Starting bucket ops to source bucket
2021/12/14 11:43:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:43:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:43:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:43:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:43:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:43:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:43:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:43:54 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:43:54 App: TestCasDelete got un-deployed
2021/12/14 11:43:54 No eventing-con process running
2021/12/14 11:43:54 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 11:43:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:44:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:44:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:44:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:44:08 Testing on the destination bucket for delete with cas
2021/12/14 11:44:08 Starting bucket ops to source bucket
2021/12/14 11:44:08 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:44:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 11:44:15 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:44:20 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:44:25 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:44:30 Waiting for app: TestCasDelete to get deployed
2021/12/14 11:44:30 App: TestCasDelete got deployed
2021/12/14 11:44:30 Starting bucket ops to source bucket
2021/12/14 11:44:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:44:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:44:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:44:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:44:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:44:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 11:44:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:44:50 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:44:55 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:45:00 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:45:05 Waiting for app: TestCasDelete to get un-deployed
2021/12/14 11:45:05 App: TestCasDelete got un-deployed
2021/12/14 11:45:05 No eventing-con process running
2021/12/14 11:45:05 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/14 11:45:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:45:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:45:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.83s)
=== RUN   TestCountersIncrement
2021/12/14 11:45:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 11:45:29 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 11:45:30 Removing node(s): [] from the cluster
2021/12/14 11:45:35 Rebalance progress: 100
2021/12/14 11:45:35 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:45:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 11:45:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:45:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:45:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:45:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:45:56 App: TestCountersIncrement got deployed
2021/12/14 11:45:56 Starting bucket ops to source bucket
2021/12/14 11:46:03 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 11:46:08 Waiting for dst bucket item count to get to: 1024 curr count: 324
2021/12/14 11:46:13 Waiting for dst bucket item count to get to: 1024 curr count: 836
2021/12/14 11:46:13 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 11:46:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:46:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:46:18 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:46:23 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:46:28 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:46:33 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:46:33 App: TestCountersIncrement got un-deployed
2021/12/14 11:46:33 No eventing-con process running
2021/12/14 11:46:33 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 11:46:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:46:45 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/14 11:46:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:46:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:46:56 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:46:56 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:46:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/14 11:47:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:47:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:47:13 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:47:18 Waiting for app: TestCountersIncrement to get deployed
2021/12/14 11:47:18 App: TestCountersIncrement got deployed
2021/12/14 11:47:18 Starting bucket ops to source bucket
2021/12/14 11:47:24 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 11:47:29 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/14 11:47:34 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/14 11:47:34 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 11:47:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:47:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:47:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:47:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:47:49 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:47:54 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/14 11:47:54 App: TestCountersIncrement got un-deployed
2021/12/14 11:47:54 No eventing-con process running
2021/12/14 11:47:54 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/14 11:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:48:06 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 11:48:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:48:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:48:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 11:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:48:17 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 11:48:23 Rebalance progress: 100
--- PASS: TestCountersIncrement (179.05s)
=== RUN   TestCountersDecrement
2021/12/14 11:48:23 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 11:48:23 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 11:48:23 Removing node(s): [] from the cluster
2021/12/14 11:48:28 Rebalance progress: 100
2021/12/14 11:48:28 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:48:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 11:48:34 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:48:39 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:48:44 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:48:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:48:49 App: TestCountersDecrement got deployed
2021/12/14 11:48:49 Starting bucket ops to source bucket
2021/12/14 11:48:56 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 11:49:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/14 11:49:06 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/14 11:49:06 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/14 11:49:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:49:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:49:11 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:49:16 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:49:21 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:49:26 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:49:26 App: TestCountersDecrement got un-deployed
2021/12/14 11:49:26 No eventing-con process running
2021/12/14 11:49:26 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 11:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:49:38 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/14 11:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:49:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:49:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 11:49:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/14 11:49:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:49:54 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:49:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/14 11:50:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:50:06 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:50:11 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:50:16 Waiting for app: TestCountersDecrement to get deployed
2021/12/14 11:50:16 App: TestCountersDecrement got deployed
2021/12/14 11:50:16 Starting bucket ops to source bucket
2021/12/14 11:50:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/14 11:50:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/14 11:50:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:50:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:50:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:50:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:50:37 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:50:42 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/14 11:50:42 App: TestCountersDecrement got un-deployed
2021/12/14 11:50:42 No eventing-con process running
2021/12/14 11:50:42 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/14 11:50:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:50:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/14 11:50:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/14 11:50:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:51:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:51:11 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 11:51:16 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 11:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:51:16 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/14 11:51:21 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.26s)
=== RUN   TestCacheGet
2021/12/14 11:51:21 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:51:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/14 11:51:27 Waiting for app: TestCacheGet to get deployed
2021/12/14 11:51:32 Waiting for app: TestCacheGet to get deployed
2021/12/14 11:51:37 Waiting for app: TestCacheGet to get deployed
2021/12/14 11:51:42 Waiting for app: TestCacheGet to get deployed
2021/12/14 11:51:42 App: TestCacheGet got deployed
2021/12/14 11:51:42 Starting bucket ops to source bucket
2021/12/14 11:51:47 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 11:51:52 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/14 11:51:57 Waiting for dst bucket item count to get to: 20 curr count: 4
2021/12/14 11:52:02 Waiting for dst bucket item count to get to: 20 curr count: 4
2021/12/14 11:52:07 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/14 11:52:12 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/14 11:52:12 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/14 11:52:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:52:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:52:17 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 11:52:22 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 11:52:27 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 11:52:32 Waiting for app: TestCacheGet to get un-deployed
2021/12/14 11:52:32 App: TestCacheGet got un-deployed
2021/12/14 11:52:32 No eventing-con process running
2021/12/14 11:52:32 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/14 11:52:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:52:44 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 11:52:49 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/14 11:52:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:52:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:53:01 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 11:53:06 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/14 11:53:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.16s)
=== RUN   TestCacheRYOW
2021/12/14 11:53:06 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:53:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/14 11:53:12 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 11:53:17 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 11:53:22 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 11:53:27 Waiting for app: TestCacheRYOW to get deployed
2021/12/14 11:53:27 App: TestCacheRYOW got deployed
2021/12/14 11:53:27 Starting bucket ops to source bucket
2021/12/14 11:53:32 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/14 11:53:37 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/14 11:53:42 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/14 11:53:42 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/14 11:53:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:53:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:53:47 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 11:53:52 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 11:53:57 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 11:54:02 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/14 11:54:02 App: TestCacheRYOW got un-deployed
2021/12/14 11:54:02 No eventing-con process running
2021/12/14 11:54:03 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/14 11:54:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:54:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:54:19 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/14 11:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:54:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:54:31 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:54:36 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (90.13s)
=== RUN   TestCacheExpiry
2021/12/14 11:54:36 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:54:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/14 11:54:42 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 11:54:47 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 11:54:52 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 11:54:57 Waiting for app: TestCacheExpiry to get deployed
2021/12/14 11:54:57 App: TestCacheExpiry got deployed
2021/12/14 11:54:57 Starting bucket ops to source bucket
2021/12/14 11:55:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:55:08 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/14 11:55:13 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/14 11:55:18 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/14 11:55:23 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/14 11:55:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:55:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:55:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:55:28 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 11:55:33 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 11:55:38 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 11:55:43 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/14 11:55:43 App: TestCacheExpiry got un-deployed
2021/12/14 11:55:43 No eventing-con process running
2021/12/14 11:55:43 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/14 11:55:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:55:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:55:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:55:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:56:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:56:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.20s)
=== RUN   TestCacheExpiryCustom
2021/12/14 11:56:06 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:56:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 11:56:12 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 11:56:18 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 11:56:23 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 11:56:28 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/14 11:56:28 App: TestCacheExpiryCustom got deployed
2021/12/14 11:56:28 Starting bucket ops to source bucket
2021/12/14 11:56:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 11:56:38 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/14 11:56:43 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/14 11:56:48 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/14 11:56:53 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/14 11:56:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 11:56:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:56:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:56:58 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 11:57:03 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 11:57:08 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 11:57:13 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/14 11:57:13 App: TestCacheExpiryCustom got un-deployed
2021/12/14 11:57:13 No eventing-con process running
2021/12/14 11:57:13 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/14 11:57:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:57:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/14 11:57:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:57:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:57:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 11:57:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.33s)
=== RUN   TestCacheOverflowCustom
2021/12/14 11:57:37 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:57:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/14 11:57:43 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 11:57:48 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 11:57:53 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 11:57:58 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/14 11:57:58 App: TestCacheOverflowCustom got deployed
2021/12/14 11:57:58 Starting bucket ops to source bucket
2021/12/14 11:58:03 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/14 11:58:08 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/14 11:58:13 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/14 11:58:18 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/14 11:58:23 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/14 11:58:28 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/14 11:58:33 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/14 11:58:33 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/14 11:58:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:58:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:58:38 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 11:58:43 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 11:58:48 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 11:58:53 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/14 11:58:53 App: TestCacheOverflowCustom got un-deployed
2021/12/14 11:58:53 No eventing-con process running
2021/12/14 11:58:53 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/14 11:58:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:59:05 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/14 11:59:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 11:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 11:59:17 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/14 11:59:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.21s)
=== RUN   TestStrictMode
2021/12/14 11:59:17 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:59:18 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/14 11:59:18 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/14 11:59:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/14 11:59:24 Waiting for app: TestDataTypes to get deployed
2021/12/14 11:59:29 Waiting for app: TestDataTypes to get deployed
2021/12/14 11:59:34 Waiting for app: TestDataTypes to get deployed
2021/12/14 11:59:39 Waiting for app: TestDataTypes to get deployed
2021/12/14 11:59:39 App: TestDataTypes got deployed
2021/12/14 11:59:39 Starting bucket ops to source bucket
2021/12/14 11:59:44 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/14 11:59:49 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/14 11:59:49 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/14 11:59:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 11:59:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 11:59:54 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 11:59:59 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 12:00:04 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 12:00:09 Waiting for app: TestDataTypes to get un-deployed
2021/12/14 12:00:09 App: TestDataTypes got un-deployed
2021/12/14 12:00:09 No eventing-con process running
2021/12/14 12:00:09 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/14 12:00:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:00:21 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 12:00:26 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 12:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:00:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:00:38 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/14 12:00:43 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 12:00:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.02s)
=== RUN   TestEnoentGet
2021/12/14 12:00:43 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:00:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 12:00:49 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:00:54 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:00:59 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:01:04 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:01:04 App: TestEnoentGet got deployed
2021/12/14 12:01:04 Starting bucket ops to source bucket
2021/12/14 12:01:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:01:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 12:01:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 12:01:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:01:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:01:19 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:01:24 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:01:29 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:01:34 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:01:34 App: TestEnoentGet got un-deployed
2021/12/14 12:01:34 No eventing-con process running
2021/12/14 12:01:34 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 12:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:01:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:01:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:01:58 Waiting for dst bucket item count to get to: 0 curr count: 66
2021/12/14 12:02:03 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/14 12:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:02:03 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:02:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/14 12:02:09 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:02:14 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:02:19 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:02:24 Waiting for app: TestEnoentGet to get deployed
2021/12/14 12:02:24 App: TestEnoentGet got deployed
2021/12/14 12:02:24 Starting bucket ops to source bucket
2021/12/14 12:02:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:02:34 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 12:02:34 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 12:02:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:02:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:02:39 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:02:45 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:02:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:02:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/14 12:02:55 App: TestEnoentGet got un-deployed
2021/12/14 12:02:55 No eventing-con process running
2021/12/14 12:02:55 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/14 12:02:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:03:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:03:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:03:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:03:18 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/14 12:03:23 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/14 12:03:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.27s)
=== RUN   TestEnoentDelete
2021/12/14 12:03:23 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:03:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 12:03:29 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:03:34 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:03:39 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:03:44 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:03:44 App: TestEnoentDelete got deployed
2021/12/14 12:03:44 Starting bucket ops to source bucket
2021/12/14 12:03:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:03:55 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 12:03:55 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 12:03:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:03:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:04:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:04:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:04:10 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:04:15 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:04:15 App: TestEnoentDelete got un-deployed
2021/12/14 12:04:15 No eventing-con process running
2021/12/14 12:04:15 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 12:04:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:04:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:04:38 Waiting for dst bucket item count to get to: 0 curr count: 77
2021/12/14 12:04:43 Waiting for dst bucket item count to get to: 0 curr count: 27
2021/12/14 12:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:04:43 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:04:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 12:04:49 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:04:54 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:04:59 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:05:04 Waiting for app: TestEnoentDelete to get deployed
2021/12/14 12:05:04 App: TestEnoentDelete got deployed
2021/12/14 12:05:04 Starting bucket ops to source bucket
2021/12/14 12:05:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:05:15 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/14 12:05:15 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 12:05:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:05:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:05:20 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:05:25 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:05:30 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:05:35 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/14 12:05:35 App: TestEnoentDelete got un-deployed
2021/12/14 12:05:35 No eventing-con process running
2021/12/14 12:05:35 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/14 12:05:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:05:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:05:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:05:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:05:58 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/14 12:06:03 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/14 12:06:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.30s)
=== RUN   TestError
2021/12/14 12:06:13 Waiting for app: TestError to get un-deployed
2021/12/14 12:06:13 App: TestError got un-deployed
2021/12/14 12:06:13 No eventing-con process running
2021/12/14 12:06:13 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/14 12:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:06:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:06:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:06:27 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:06:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/14 12:06:33 Waiting for app: TestError to get deployed
2021/12/14 12:06:38 Waiting for app: TestError to get deployed
2021/12/14 12:06:43 Waiting for app: TestError to get deployed
2021/12/14 12:06:48 Waiting for app: TestError to get deployed
2021/12/14 12:06:48 App: TestError got deployed
2021/12/14 12:06:48 Starting bucket ops to source bucket
2021/12/14 12:06:54 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/14 12:06:59 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/14 12:06:59 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/14 12:06:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:06:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:07:04 Waiting for app: TestError to get un-deployed
2021/12/14 12:07:09 Waiting for app: TestError to get un-deployed
2021/12/14 12:07:14 Waiting for app: TestError to get un-deployed
2021/12/14 12:07:19 Waiting for app: TestError to get un-deployed
2021/12/14 12:07:19 App: TestError got un-deployed
2021/12/14 12:07:19 No eventing-con process running
2021/12/14 12:07:19 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/14 12:07:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:07:31 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/14 12:07:36 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/14 12:07:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:07:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:07:47 Waiting for dst bucket item count to get to: 0 curr count: 144
2021/12/14 12:07:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.90s)
=== RUN   TestCRLF
2021/12/14 12:07:57 Waiting for app: TestCRLF to get un-deployed
2021/12/14 12:07:57 App: TestCRLF got un-deployed
2021/12/14 12:07:57 No eventing-con process running
2021/12/14 12:07:57 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/14 12:07:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:08:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:08:11 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:08:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/14 12:08:17 Waiting for app: TestCRLF to get deployed
2021/12/14 12:08:22 Waiting for app: TestCRLF to get deployed
2021/12/14 12:08:27 Waiting for app: TestCRLF to get deployed
2021/12/14 12:08:32 Waiting for app: TestCRLF to get deployed
2021/12/14 12:08:32 App: TestCRLF got deployed
2021/12/14 12:08:32 Starting bucket ops to source bucket
2021/12/14 12:08:37 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:08:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 12:08:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 12:08:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:08:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:08:47 Waiting for app: TestCRLF to get un-deployed
2021/12/14 12:08:52 Waiting for app: TestCRLF to get un-deployed
2021/12/14 12:08:57 Waiting for app: TestCRLF to get un-deployed
2021/12/14 12:09:02 Waiting for app: TestCRLF to get un-deployed
2021/12/14 12:09:02 App: TestCRLF got un-deployed
2021/12/14 12:09:02 No eventing-con process running
2021/12/14 12:09:03 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/14 12:09:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:09:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:09:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:09:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:09:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 12:09:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (98.71s)
=== RUN   TestImportExport
2021/12/14 12:09:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:09:36 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 12:09:36 No eventing-con process running
2021/12/14 12:09:36 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/14 12:09:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:09:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:09:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:09:55 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:09:55 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 12:09:55 No eventing-con process running
2021/12/14 12:09:55 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/14 12:09:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:10:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:10:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:10:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:10:08 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:10:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 12:10:10 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:10:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 12:10:16 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 12:10:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 12:10:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 12:10:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/14 12:10:31 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/14 12:10:36 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 12:10:41 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 12:10:46 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/14 12:10:46 App: TestImportExport_bucket_op_on_update got deployed
2021/12/14 12:10:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:10:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:10:56 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:11:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:11:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:11:06 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 12:11:06 No eventing-con process running
2021/12/14 12:11:06 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 12:11:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:11:11 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:11:16 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:11:21 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:11:26 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:11:26 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 12:11:26 No eventing-con process running
2021/12/14 12:11:26 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 12:11:43 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/14 12:11:43 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/14 12:11:43 No eventing-con process running
2021/12/14 12:11:43 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/14 12:11:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:11:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:11:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:12:02 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/14 12:12:02 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/14 12:12:02 No eventing-con process running
2021/12/14 12:12:02 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/14 12:12:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:12:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:12:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.60s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/14 12:12:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:12:26 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:12:26 No eventing-con process running
2021/12/14 12:12:26 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/14 12:12:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:12:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:12:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:12:39 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:12:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:12:40 Starting bucket ops to source bucket
2021/12/14 12:12:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:12:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:12:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:13:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:13:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:13:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 12:13:08 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 12:13:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:13:08 Undeploying app: bucket_op_on_update
2021/12/14 12:13:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:13:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:13:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:13:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:13:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:13:29 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:13:29 No eventing-con process running
2021/12/14 12:13:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:14:13 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:14:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:14:14 Starting bucket ops to source bucket
2021/12/14 12:14:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:14:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:14:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:14:37 Waiting for dst bucket item count to get to: 5000 curr count: 1911
2021/12/14 12:14:42 Waiting for dst bucket item count to get to: 5000 curr count: 4411
2021/12/14 12:14:42 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:14:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 12:14:47 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 12:14:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:14:47 Undeploying app: bucket_op_on_update
2021/12/14 12:14:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:14:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:14:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:15:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:15:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:15:07 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:15:07 No eventing-con process running
2021/12/14 12:15:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:15:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:15:51 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:15:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:15:52 Starting bucket ops to source bucket
2021/12/14 12:16:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:16:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:16:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 12:16:20 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 12:16:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:16:20 Undeploying app: bucket_op_on_update
2021/12/14 12:16:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:16:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:16:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:16:35 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:16:40 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:16:40 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:16:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:16:40 No eventing-con process running
2021/12/14 12:16:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:16:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:17:24 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:17:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:17:26 Starting bucket ops to source bucket
2021/12/14 12:17:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:17:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:17:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:17:49 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 12:17:54 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 12:17:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:17:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 12:17:59 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 12:17:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:17:59 Undeploying app: bucket_op_on_update
2021/12/14 12:17:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:18:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:18:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:18:14 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:18:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:18:19 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:18:19 No eventing-con process running
2021/12/14 12:18:20 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:18:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:19:03 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:19:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:19:04 Starting bucket ops to source bucket
2021/12/14 12:19:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:19:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:19:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:19:27 Waiting for dst bucket item count to get to: 5000 curr count: 1712
2021/12/14 12:19:32 Waiting for dst bucket item count to get to: 5000 curr count: 4212
2021/12/14 12:19:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:19:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/14 12:19:37 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/14 12:19:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:19:37 Undeploying app: bucket_op_on_update
2021/12/14 12:19:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:19:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:19:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:19:52 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:19:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/14 12:19:57 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/14 12:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:19:57 No eventing-con process running
2021/12/14 12:19:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:20:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:20:41 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (505.59s)
=== RUN   TestOnUpdateN1QLOp
2021/12/14 12:20:51 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 12:20:51 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 12:20:51 No eventing-con process running
2021/12/14 12:20:51 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/14 12:20:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:21:05 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:21:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/14 12:21:06 Starting bucket ops to source bucket
2021/12/14 12:21:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:21:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:21:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:21:29 Waiting for dst bucket item count to get to: 5000 curr count: 539
2021/12/14 12:21:34 Waiting for dst bucket item count to get to: 5000 curr count: 539
2021/12/14 12:21:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:21:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:21:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:21:39 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 12:21:44 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 12:21:49 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 12:21:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/14 12:21:54 App: TestOnUpdateN1QLOp got un-deployed
2021/12/14 12:21:54 No eventing-con process running
2021/12/14 12:21:54 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/14 12:21:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:22:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:22:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:22:18 Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/14 12:22:23 Waiting for dst bucket item count to get to: 0 curr count: 737
2021/12/14 12:22:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.66s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/14 12:22:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 12:22:33 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 12:22:33 No eventing-con process running
2021/12/14 12:22:33 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/14 12:22:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:22:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:22:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:22:47 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:22:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:22:48 Starting bucket ops to source bucket
2021/12/14 12:22:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:23:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:23:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:23:11 Waiting for dst bucket item count to get to: 5000 curr count: 388
2021/12/14 12:23:16 Waiting for dst bucket item count to get to: 5000 curr count: 388
2021/12/14 12:23:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:23:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:23:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:23:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 12:23:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 12:23:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 12:23:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/14 12:23:36 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/14 12:23:36 No eventing-con process running
2021/12/14 12:23:36 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/14 12:23:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:23:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 12:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:23:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:23:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 12:24:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 12:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.57s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/14 12:24:14 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 12:24:14 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 12:24:14 No eventing-con process running
2021/12/14 12:24:14 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 12:24:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:24:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:24:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:24:28 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:24:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:24:29 Starting bucket ops to source bucket
2021/12/14 12:24:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:24:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:24:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:24:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:24:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/14 12:24:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:24:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:24:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:25:02 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 12:25:07 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 12:25:12 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 12:25:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 12:25:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 12:25:17 No eventing-con process running
2021/12/14 12:25:17 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 12:25:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:25:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:25:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 12:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:25:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:25:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:25:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.58s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/14 12:25:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 12:25:56 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 12:25:56 No eventing-con process running
2021/12/14 12:25:56 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/14 12:25:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:26:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:26:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:26:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:26:10 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:26:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/14 12:26:11 Starting bucket ops to source bucket
2021/12/14 12:26:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:26:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:26:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:26:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:26:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:26:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:26:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:26:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 12:26:44 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 12:26:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 12:26:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/14 12:26:54 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/14 12:26:54 No eventing-con process running
2021/12/14 12:26:54 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/14 12:26:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:27:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:27:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:27:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:27:18 Waiting for dst bucket item count to get to: 0 curr count: 2808
2021/12/14 12:27:23 Waiting for dst bucket item count to get to: 0 curr count: 308
2021/12/14 12:27:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.68s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/14 12:27:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 12:27:33 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 12:27:33 No eventing-con process running
2021/12/14 12:27:33 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/14 12:27:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:27:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:27:46 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:27:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/14 12:27:48 Starting bucket ops to source bucket
2021/12/14 12:27:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:28:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:28:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:28:11 Waiting for dst bucket item count to get to: 5000 curr count: 422
2021/12/14 12:28:16 Waiting for dst bucket item count to get to: 5000 curr count: 422
2021/12/14 12:28:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:28:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:28:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:28:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 12:28:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 12:28:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 12:28:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/14 12:28:36 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/14 12:28:36 No eventing-con process running
2021/12/14 12:28:36 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/14 12:28:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:28:48 Waiting for dst bucket item count to get to: 0 curr count: 3299
2021/12/14 12:28:53 Waiting for dst bucket item count to get to: 0 curr count: 799
2021/12/14 12:28:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:28:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:29:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.12s)
=== RUN   TestOnDeleteBucketOp
2021/12/14 12:29:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 12:29:15 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 12:29:15 No eventing-con process running
2021/12/14 12:29:15 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/14 12:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:29:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:29:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:29:28 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:29:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 12:29:30 Starting bucket ops to source bucket
2021/12/14 12:29:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:29:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:29:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:29:55 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/14 12:29:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:29:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:29:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:30:01 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 12:30:06 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 12:30:11 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 12:30:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/14 12:30:16 App: TestOnDeleteBucketOp got un-deployed
2021/12/14 12:30:16 No eventing-con process running
2021/12/14 12:30:16 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/14 12:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:30:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:30:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:30:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.82s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/14 12:30:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 12:30:45 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 12:30:45 No eventing-con process running
2021/12/14 12:30:45 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/14 12:30:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:30:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:30:58 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:30:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/14 12:30:59 Starting bucket ops to source bucket
2021/12/14 12:31:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:31:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:31:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:31:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/14 12:31:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:31:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:31:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:31:31 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 12:31:36 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 12:31:41 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 12:31:46 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/14 12:31:46 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/14 12:31:46 No eventing-con process running
2021/12/14 12:31:46 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/14 12:31:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:31:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:31:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:32:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:32:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.02s)
=== RUN   TestTimerBucketOp
2021/12/14 12:32:15 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 12:32:15 App: TestTimerBucketOp got un-deployed
2021/12/14 12:32:15 No eventing-con process running
2021/12/14 12:32:15 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/14 12:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:32:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:32:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:32:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:32:28 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:32:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 12:32:29 Starting bucket ops to source bucket
2021/12/14 12:32:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:32:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:32:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:32:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:32:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:33:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:33:08 Waiting for dst bucket item count to get to: 5000 curr count: 1389
2021/12/14 12:33:13 Waiting for dst bucket item count to get to: 5000 curr count: 3889
2021/12/14 12:33:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:33:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:33:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:33:18 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 12:33:23 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 12:33:28 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 12:33:33 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/14 12:33:33 App: TestTimerBucketOp got un-deployed
2021/12/14 12:33:33 No eventing-con process running
2021/12/14 12:33:33 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/14 12:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:33:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:33:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:33:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:33:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.59s)
=== RUN   TestTimerInPastBucketOp
2021/12/14 12:34:06 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 12:34:06 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 12:34:06 No eventing-con process running
2021/12/14 12:34:06 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/14 12:34:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:34:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:34:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:34:20 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:34:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 12:34:21 Starting bucket ops to source bucket
2021/12/14 12:34:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:34:59 Waiting for dst bucket item count to get to: 5000 curr count: 1922
2021/12/14 12:35:04 Waiting for dst bucket item count to get to: 5000 curr count: 1922
2021/12/14 12:35:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:35:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:35:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:35:09 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 12:35:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 12:35:19 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 12:35:24 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/14 12:35:24 App: TestTimerInPastBucketOp got un-deployed
2021/12/14 12:35:24 No eventing-con process running
2021/12/14 12:35:24 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/14 12:35:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:35:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 12:35:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:35:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:35:48 Waiting for dst bucket item count to get to: 0 curr count: 3240
2021/12/14 12:35:53 Waiting for dst bucket item count to get to: 0 curr count: 740
2021/12/14 12:35:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (116.58s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/14 12:36:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:36:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:36:03 No eventing-con process running
2021/12/14 12:36:03 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/14 12:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:36:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:36:17 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:36:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 12:36:18 Starting bucket ops to source bucket
2021/12/14 12:36:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:36:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:36:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:36:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:36:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:36:51 Waiting for dst bucket item count to get to: 5000 curr count: 1271
2021/12/14 12:36:56 Waiting for dst bucket item count to get to: 5000 curr count: 1271
2021/12/14 12:36:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:37:01 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 12:37:01 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 12:37:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:37:01 Undeploying app: bucket_op_with_timer
2021/12/14 12:37:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:37:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:37:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:37:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:37:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:37:21 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:37:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:37:21 No eventing-con process running
2021/12/14 12:37:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:37:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:38:05 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:38:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 12:38:06 Starting bucket ops to source bucket
2021/12/14 12:38:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:38:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:38:49 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 12:38:49 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 12:38:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:38:49 Undeploying app: bucket_op_with_timer
2021/12/14 12:38:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:38:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:38:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:39:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:39:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:39:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:39:09 No eventing-con process running
2021/12/14 12:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:39:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:39:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:39:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 12:39:54 Starting bucket ops to source bucket
2021/12/14 12:40:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:40:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:40:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:40:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:40:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:40:27 Waiting for dst bucket item count to get to: 5000 curr count: 994
2021/12/14 12:40:32 Waiting for dst bucket item count to get to: 5000 curr count: 2804
2021/12/14 12:40:37 Waiting for dst bucket item count to get to: 5000 curr count: 4310
2021/12/14 12:40:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:40:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 12:40:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 12:40:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:40:42 Undeploying app: bucket_op_with_timer
2021/12/14 12:40:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:40:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:40:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:40:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:41:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:41:03 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:41:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:41:03 No eventing-con process running
2021/12/14 12:41:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:41:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:41:47 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:41:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 12:41:48 Starting bucket ops to source bucket
2021/12/14 12:41:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:42:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:42:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:42:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:42:16 Waiting for dst bucket item count to get to: 5000 curr count: 12
2021/12/14 12:42:21 Waiting for dst bucket item count to get to: 5000 curr count: 284
2021/12/14 12:42:26 Waiting for dst bucket item count to get to: 5000 curr count: 2772
2021/12/14 12:42:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:42:31 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 12:42:31 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 12:42:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:42:31 Undeploying app: bucket_op_with_timer
2021/12/14 12:42:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:42:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:42:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:42:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:42:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:42:51 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:42:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:42:51 No eventing-con process running
2021/12/14 12:42:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:43:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:43:35 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:43:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 12:43:36 Starting bucket ops to source bucket
2021/12/14 12:43:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:43:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:43:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:43:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:44:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:44:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:44:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:44:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:44:19 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/14 12:44:19 App: TestDeployUndeployLoopTimer got deployed
2021/12/14 12:44:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:44:19 Undeploying app: bucket_op_with_timer
2021/12/14 12:44:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:44:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:44:30 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:44:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:44:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/14 12:44:40 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/14 12:44:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:44:40 No eventing-con process running
2021/12/14 12:44:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:44:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:45:24 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (570.78s)
=== RUN   TestMultipleHandlers
2021/12/14 12:45:34 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 12:45:34 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 12:45:34 No eventing-con process running
2021/12/14 12:45:34 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/14 12:45:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:45:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:45:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:45:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:45:52 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 12:45:52 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 12:45:52 No eventing-con process running
2021/12/14 12:45:52 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/14 12:45:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:45:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:46:01 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:46:06 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 12:46:07 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:46:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 12:46:13 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 12:46:18 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 12:46:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 12:46:28 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/14 12:46:28 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/14 12:46:33 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 12:46:38 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 12:46:43 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/14 12:46:43 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/14 12:46:43 Starting bucket ops to source bucket
2021/12/14 12:46:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:46:58 Waiting for dst bucket item count to get to: 5000 curr count: 4503
2021/12/14 12:47:03 Waiting for dst bucket item count to get to: 5000 curr count: 4503
2021/12/14 12:47:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:47:03 Starting bucket ops to source bucket
2021/12/14 12:47:18 Waiting for dst bucket item count to get to: 10000 curr count: 5564
2021/12/14 12:47:23 Waiting for dst bucket item count to get to: 10000 curr count: 5564
2021/12/14 12:47:23 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 12:47:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:47:23 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:47:23 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:47:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:47:29 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 12:47:34 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 12:47:39 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 12:47:44 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/14 12:47:44 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/14 12:47:44 No eventing-con process running
2021/12/14 12:47:44 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/14 12:47:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:47:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:47:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:48:03 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 12:48:08 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/14 12:48:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:48:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:48:13 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 12:48:18 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 12:48:23 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 12:48:28 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/14 12:48:28 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/14 12:48:28 No eventing-con process running
2021/12/14 12:48:28 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/14 12:48:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:48:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:48:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:48:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.02s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/14 12:48:52 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 12:48:52 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 12:48:52 No eventing-con process running
2021/12/14 12:48:52 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/14 12:48:53 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:48:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:49:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:49:05 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:49:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:49:06 Starting bucket ops to source bucket
2021/12/14 12:49:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:49:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:49:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:49:29 Waiting for dst bucket item count to get to: 5000 curr count: 1963
2021/12/14 12:49:34 Waiting for dst bucket item count to get to: 5000 curr count: 1963
2021/12/14 12:49:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:49:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:49:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 12:49:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 12:49:39 Pausing app: bucket_op_on_update
2021/12/14 12:49:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:49:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 12:49:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 12:49:45 Resuming app: bucket_op_on_update
2021/12/14 12:49:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:49:46 Starting bucket ops to source bucket
2021/12/14 12:49:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:49:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:50:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:50:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 12:50:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:50:09 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 12:50:09 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 12:50:09 Pausing app: bucket_op_on_update
2021/12/14 12:50:09 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:50:14 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 12:50:14 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 12:50:14 Resuming app: bucket_op_on_update
2021/12/14 12:50:15 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:50:15 Starting bucket ops to source bucket
2021/12/14 12:50:23 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:50:28 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:50:33 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:50:33 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 12:50:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:50:38 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 12:50:38 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 12:50:38 Pausing app: bucket_op_on_update
2021/12/14 12:50:38 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:50:43 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 12:50:43 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 12:50:43 Resuming app: bucket_op_on_update
2021/12/14 12:50:44 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:50:44 Starting bucket ops to source bucket
2021/12/14 12:50:52 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:50:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:51:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:51:03 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 12:51:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:51:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 12:51:08 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 12:51:08 Pausing app: bucket_op_on_update
2021/12/14 12:51:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:51:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 12:51:13 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 12:51:13 Resuming app: bucket_op_on_update
2021/12/14 12:51:14 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:51:14 Starting bucket ops to source bucket
2021/12/14 12:51:22 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:51:27 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:51:32 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:51:37 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/14 12:51:37 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 12:51:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:51:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/14 12:51:42 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/14 12:51:42 Pausing app: bucket_op_on_update
2021/12/14 12:51:42 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:51:47 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/14 12:51:47 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/14 12:51:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:51:52 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 12:51:57 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 12:52:02 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 12:52:07 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/14 12:52:07 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/14 12:52:07 No eventing-con process running
2021/12/14 12:52:07 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/14 12:52:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:52:19 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 12:52:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 12:52:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:52:36 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 12:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.20s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/14 12:52:46 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 12:52:46 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 12:52:46 No eventing-con process running
2021/12/14 12:52:46 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/14 12:52:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:52:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:52:59 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:53:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 12:53:01 Starting bucket ops to source bucket
2021/12/14 12:53:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:53:39 Waiting for dst bucket item count to get to: 5000 curr count: 2797
2021/12/14 12:53:44 Waiting for dst bucket item count to get to: 5000 curr count: 2797
2021/12/14 12:53:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:53:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:53:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 12:53:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 12:53:49 Pausing app: bucket_op_with_timer
2021/12/14 12:53:49 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:53:54 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 12:53:54 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 12:53:54 Resuming app: bucket_op_with_timer
2021/12/14 12:53:55 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:53:55 Starting bucket ops to source bucket
2021/12/14 12:54:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:54:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:54:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:54:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:54:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:54:28 Waiting for dst bucket item count to get to: 10000 curr count: 8913
2021/12/14 12:54:33 Waiting for dst bucket item count to get to: 10000 curr count: 8913
2021/12/14 12:54:33 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 12:54:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:54:38 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 12:54:38 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 12:54:38 Pausing app: bucket_op_with_timer
2021/12/14 12:54:38 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:54:43 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 12:54:43 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 12:54:43 Resuming app: bucket_op_with_timer
2021/12/14 12:54:44 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:54:44 Starting bucket ops to source bucket
2021/12/14 12:54:52 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:54:57 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:55:02 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:55:07 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:55:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/14 12:55:17 Waiting for dst bucket item count to get to: 15000 curr count: 14850
2021/12/14 12:55:22 Waiting for dst bucket item count to get to: 15000 curr count: 14850
2021/12/14 12:55:22 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/14 12:55:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:55:27 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 12:55:27 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 12:55:27 Pausing app: bucket_op_with_timer
2021/12/14 12:55:28 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:55:33 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 12:55:33 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 12:55:33 Resuming app: bucket_op_with_timer
2021/12/14 12:55:34 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:55:34 Starting bucket ops to source bucket
2021/12/14 12:55:42 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:55:47 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:55:52 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:55:57 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:56:02 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:56:07 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/14 12:56:07 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 12:56:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:56:12 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 12:56:12 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 12:56:12 Pausing app: bucket_op_with_timer
2021/12/14 12:56:12 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:56:17 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 12:56:17 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 12:56:17 Resuming app: bucket_op_with_timer
2021/12/14 12:56:18 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:56:18 Starting bucket ops to source bucket
2021/12/14 12:56:26 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:56:31 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:56:36 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:56:41 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:56:46 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/14 12:56:51 Waiting for dst bucket item count to get to: 25000 curr count: 20223
2021/12/14 12:56:56 Waiting for dst bucket item count to get to: 25000 curr count: 22723
2021/12/14 12:56:56 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 12:56:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:57:01 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/14 12:57:01 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/14 12:57:01 Pausing app: bucket_op_with_timer
2021/12/14 12:57:01 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:57:06 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/14 12:57:06 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/14 12:57:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 12:57:11 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 12:57:16 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 12:57:21 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 12:57:26 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/14 12:57:26 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/14 12:57:26 No eventing-con process running
2021/12/14 12:57:26 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/14 12:57:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:57:38 Waiting for dst bucket item count to get to: 0 curr count: 17183
2021/12/14 12:57:43 Waiting for dst bucket item count to get to: 0 curr count: 4683
2021/12/14 12:57:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:57:55 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 12:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (318.97s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/14 12:58:05 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 12:58:05 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 12:58:05 No eventing-con process running
2021/12/14 12:58:05 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/14 12:58:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:58:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:58:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 12:58:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 12:58:18 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/14 12:58:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/14 12:58:25 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 12:58:30 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 12:58:35 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 12:58:40 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/14 12:58:40 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/14 12:58:40 Starting bucket ops to source bucket
2021/12/14 12:58:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:58:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:58:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:59:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 12:59:09 Waiting for dst bucket item count to get to: 5000 curr count: 508
2021/12/14 12:59:14 Waiting for dst bucket item count to get to: 5000 curr count: 1014
2021/12/14 12:59:19 Waiting for dst bucket item count to get to: 5000 curr count: 3006
2021/12/14 12:59:19 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 12:59:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 12:59:19 Pausing app: bucket_op_with_timer
2021/12/14 12:59:19 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:59:24 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/14 12:59:24 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/14 12:59:24 Resuming app: bucket_op_with_timer
2021/12/14 12:59:25 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 12:59:25 Starting bucket ops to source bucket
2021/12/14 12:59:37 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:59:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:59:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:59:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 12:59:57 Waiting for dst bucket item count to get to: 10000 curr count: 7275
2021/12/14 13:00:02 Waiting for dst bucket item count to get to: 10000 curr count: 7275
2021/12/14 13:00:02 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 13:00:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:00:07 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 13:00:12 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 13:00:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 13:00:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/14 13:00:22 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/14 13:00:22 No eventing-con process running
2021/12/14 13:00:22 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/14 13:00:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:00:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:00:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:00:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:00:50 Waiting for dst bucket item count to get to: 0 curr count: 8208
2021/12/14 13:00:55 Waiting for dst bucket item count to get to: 0 curr count: 8208
2021/12/14 13:00:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (180.63s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/14 13:01:05 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 13:01:05 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 13:01:05 No eventing-con process running
2021/12/14 13:01:06 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/14 13:01:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:01:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:01:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:01:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:01:19 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:01:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 13:01:25 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 13:01:30 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 13:01:35 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 13:01:40 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/14 13:01:40 App: TestPauseResumeWithEventingReb got deployed
2021/12/14 13:01:40 Starting bucket ops to source bucket
2021/12/14 13:01:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:01:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:02:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:02:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:02:10 Waiting for dst bucket item count to get to: 5000 curr count: 2603
2021/12/14 13:02:15 Waiting for dst bucket item count to get to: 5000 curr count: 2603
2021/12/14 13:02:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:02:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:02:15 Pausing app: bucket_op_with_timer
2021/12/14 13:02:15 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:02:20 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/14 13:02:20 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/14 13:02:20 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 13:02:20 Removing node(s): [] from the cluster
2021/12/14 13:02:25 Rebalance progress: 29.16666666666666
2021/12/14 13:02:30 Rebalance progress: 29.16666666666666
2021/12/14 13:02:35 Rebalance progress: 100
2021/12/14 13:02:35 Resuming app: bucket_op_with_timer
2021/12/14 13:02:36 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:02:36 Starting bucket ops to source bucket
2021/12/14 13:02:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 13:02:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 13:02:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 13:03:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/14 13:03:08 Waiting for dst bucket item count to get to: 10000 curr count: 5859
2021/12/14 13:03:13 Waiting for dst bucket item count to get to: 10000 curr count: 5859
2021/12/14 13:03:18 Waiting for dst bucket item count to get to: 10000 curr count: 9964
2021/12/14 13:03:23 Waiting for dst bucket item count to get to: 10000 curr count: 9964
2021/12/14 13:03:23 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 13:03:23 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 13:03:28 Rebalance progress: 29.16666666666666
2021/12/14 13:03:33 Rebalance progress: 29.16666666666666
2021/12/14 13:03:38 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 13:03:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:03:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:04:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:04:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:04:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:04:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:05:14 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:05:29 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:05:44 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 13:05:59 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:59} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 55dc35816e34b0d44cca874a1c0a51e3
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
Producer node: 172.31.2.134:9300	Node UUID: dc8924a14720ee86963beef25c033c83
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 4e3cbb387e6b356ac9e7a88d768262d3
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/14 13:06:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:06:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:06:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 13:06:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 13:06:29 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 13:06:34 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/14 13:06:34 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/14 13:06:34 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:06:34 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/14 13:06:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:06:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:06:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:06:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:06:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:06:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (362.19s)
=== RUN   TestBucketDeleteAfterPause
2021/12/14 13:07:08 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 13:07:08 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 13:07:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:07:08 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/14 13:07:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:07:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:07:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:07:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:07:21 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:07:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/14 13:07:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 13:07:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 13:07:38 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 13:07:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/14 13:07:43 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/14 13:07:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:07:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:07:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:07:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:07:52 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/14 13:07:52 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:07:57 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/14 13:07:57 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/14 13:07:57 handler_test.go:716: Deleting source bucket: default
2021/12/14 13:08:04 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/14 13:08:04 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/14 13:08:09 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/14 13:08:09 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/14 13:08:09 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:08:09 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/14 13:08:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:08:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:08:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:08:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:08:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.44s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/14 13:08:37 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 13:08:37 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 13:08:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:08:37 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/14 13:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:08:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:08:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:08:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:08:50 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:08:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 13:08:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 13:09:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 13:09:06 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 13:09:11 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/14 13:09:11 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/14 13:09:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:09:46 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/14 13:09:46 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:09:51 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/14 13:09:51 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/14 13:12:51 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/14 13:12:51 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/14 13:12:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/14 13:12:58 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 13:12:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:12:58 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/14 13:13:03 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 13:13:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:13:03 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3131 expected 5000
2021/12/14 13:13:08 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 13:13:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:13:08 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/14 13:13:13 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/14 13:13:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:13:13 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/14 13:13:13 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/14 13:13:13 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:13:23 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 13:13:28 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 13:13:33 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/14 13:13:33 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/14 13:13:33 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:13:33 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/14 13:13:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:13:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:13:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 13:13:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:13:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:13:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.67s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/14 13:14:06 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 13:14:06 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 13:14:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:14:06 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/14 13:14:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:14:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:14:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:14:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:14:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:14:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 13:14:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:14:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:14:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:14:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:14:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 13:14:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:14:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:14:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:14:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:14:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:14:54 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 13:14:54 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:15:24 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/14 13:15:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 13:15:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:15:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:15:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:15:45 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/14 13:15:45 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/14 13:15:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:15:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:16:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:16:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:16:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:16:03 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/14 13:16:03 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:16:13 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 13:16:18 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 13:16:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/14 13:16:23 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/14 13:16:23 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:16:23 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/14 13:16:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:16:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:16:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:16:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.56s)
=== RUN   TestCPPWorkerCleanup
2021/12/14 13:16:46 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 13:16:46 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 13:16:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:16:46 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/14 13:16:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:16:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:16:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:17:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:17:00 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:17:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/14 13:17:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:17:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:17:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:17:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:17:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:17:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4453
2021/12/14 13:17:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4453
2021/12/14 13:17:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:17:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:17:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:17:38 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 13:17:43 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 13:17:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 13:17:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/14 13:17:53 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/14 13:17:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:17:53 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/14 13:17:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:18:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:18:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 13:18:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:18:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:18:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:18:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:18:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.71s)
=== RUN   TestWithUserXattrs
2021/12/14 13:19:06 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 13:19:06 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 13:19:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:19:06 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/14 13:19:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:19:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:19:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:19:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:19:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:19:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:19:23 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:19:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/14 13:19:29 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 13:19:34 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 13:19:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 13:19:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/14 13:19:44 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/14 13:19:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:19:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:19:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4052
2021/12/14 13:20:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4052
2021/12/14 13:20:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:20:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:20:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:20:09 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 13:20:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 13:20:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 13:20:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/14 13:20:24 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/14 13:20:24 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:20:24 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/14 13:20:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:20:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:20:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:20:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:20:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:20:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.98s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/14 13:20:57 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:20:57 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 13:20:57 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:20:57 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/14 13:20:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:21:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:21:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:21:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:21:10 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:21:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/14 13:21:16 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 13:21:21 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 13:21:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 13:21:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/14 13:21:31 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/14 13:21:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:21:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:21:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:21:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:21:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:21:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:21:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3322
2021/12/14 13:22:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 822
2021/12/14 13:22:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:22:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:22:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:22:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:22:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:22:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15275
2021/12/14 13:22:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15275
2021/12/14 13:22:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/14 13:22:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:22:46 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:22:51 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:22:56 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:23:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:23:06 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:23:06 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 13:23:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:23:11 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/14 13:23:11 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/14 13:23:11 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:23:11 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/14 13:23:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:23:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/14 13:23:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 13:23:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:23:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:23:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 13:23:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/14 13:23:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.38s)
=== RUN   TestMetaBucketDelete
2021/12/14 13:23:54 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 13:23:54 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 13:23:54 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:23:54 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/14 13:23:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:24:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:24:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:24:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:24:08 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:24:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 13:24:14 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 13:24:19 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 13:24:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 13:24:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/14 13:24:29 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/14 13:24:29 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/14 13:24:30 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/14 13:24:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 13:24:35 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 13:24:50 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/14 13:24:50 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/14 13:24:50 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:24:50 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/14 13:24:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:24:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:24:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:25:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.41s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/14 13:25:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:25:13 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:25:13 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:25:13 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 13:25:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:25:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:25:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:25:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:25:26 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:25:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 13:25:46 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/14 13:25:46 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/14 13:26:01 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:26:02 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:26:17 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:26:17 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:26:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:26:17 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 13:26:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:26:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:26:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:26:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.78s)
=== RUN   TestSourceBucketDelete
2021/12/14 13:26:40 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 13:26:40 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 13:26:40 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:26:40 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/14 13:26:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:26:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:26:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:26:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:26:53 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:26:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/14 13:26:59 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 13:27:04 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 13:27:09 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 13:27:14 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/14 13:27:14 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/14 13:27:14 handler_test.go:945: Deleting source bucket: default
2021/12/14 13:27:16 handler_test.go:947: Deleted source bucket: default
2021/12/14 13:27:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 13:27:21 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 13:27:36 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/14 13:27:36 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/14 13:27:36 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:27:36 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/14 13:27:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:27:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:27:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:27:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.31s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/14 13:27:59 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:27:59 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:27:59 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:27:59 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 13:28:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:28:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:28:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:28:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:28:12 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:28:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 13:28:32 handler_test.go:966: Deleting source bucket: default
2021/12/14 13:28:33 handler_test.go:968: Deleted source bucket: default
2021/12/14 13:28:48 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:28:48 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:29:03 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:29:03 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:29:03 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:29:03 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 13:29:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:29:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:29:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:29:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.66s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/14 13:29:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:29:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:29:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:29:27 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/14 13:29:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:29:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:29:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:29:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:29:40 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:29:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 13:30:00 handler_test.go:988: Deleting source bucket: default
2021/12/14 13:30:01 handler_test.go:990: Deleted source bucket: default
2021/12/14 13:30:01 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/14 13:30:02 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/14 13:30:17 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:30:17 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:30:32 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/14 13:30:32 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/14 13:30:32 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:30:32 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/14 13:30:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:30:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:30:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:30:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.18s)
=== RUN   TestUndeployDuringBootstrap
2021/12/14 13:30:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 13:30:55 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 13:30:55 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:30:55 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/14 13:30:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:31:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:31:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:31:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:31:08 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:31:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 13:31:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:31:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:31:21 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:31:21 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 13:31:21 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/14 13:31:21 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/14 13:31:26 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/14 13:31:31 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 13:31:36 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 13:31:41 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 13:31:41 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 13:31:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:32:06 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/14 13:32:06 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/14 13:32:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:32:06 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/14 13:32:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:32:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 13:32:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:32:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:32:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 316
2021/12/14 13:32:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 316
2021/12/14 13:32:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.66s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/14 13:32:39 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:32:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/14 13:32:46 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 13:32:51 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 13:32:56 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 13:33:01 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/14 13:33:01 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/14 13:33:01 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/14 13:33:01 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:33:01 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/14 13:33:06 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/14 13:33:06 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/14 13:33:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:33:06 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/14 13:33:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:33:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:33:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:33:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.67s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/14 13:33:29 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:33:29 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 13:33:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:33:29 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/14 13:33:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:33:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:33:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:33:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:33:42 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:33:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/14 13:33:49 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 13:33:54 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 13:33:59 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 13:34:04 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/14 13:34:04 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/14 13:34:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:34:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:34:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:34:44 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:34:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:34:54 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:34:54 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 13:34:54 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:36:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/14 13:36:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/14 13:36:39 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:36:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/14 13:36:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:36:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4271
2021/12/14 13:36:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4271
2021/12/14 13:36:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:36:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:37:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:37:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.05s)
=== RUN   TestUndeployWithKVFailover
2021/12/14 13:37:17 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 13:37:17 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/14 13:37:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 13:37:17 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/14 13:37:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:37:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:37:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:37:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:37:30 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:37:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/14 13:37:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 13:37:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:37:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:37:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:37:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:37:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:38:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:38:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1525
2021/12/14 13:38:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1525
2021/12/14 13:38:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:38:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 13:38:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 13:38:19 Rebalance progress: 29.16666666666666
2021/12/14 13:38:24 Rebalance progress: 29.16666666666666
2021/12/14 13:38:29 Rebalance progress: 100
2021/12/14 13:38:29 Failing over: [127.0.0.1:19003]
2021/12/14 13:38:40 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/14 13:39:40 Removing node(s): [] from the cluster
2021/12/14 13:39:45 Rebalance progress: 29.16666666666666
2021/12/14 13:39:50 Rebalance progress: 29.16666666666666
2021/12/14 13:39:55 Rebalance progress: 100
2021/12/14 13:40:00 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 13:40:00 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 13:40:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:40:05 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/14 13:40:05 App: TestUndeployWithKVFailover got un-deployed
2021/12/14 13:40:05 No eventing-con process running
2021/12/14 13:40:05 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/14 13:40:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:40:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:40:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:40:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:40:28 Waiting for dst bucket item count to get to: 0 curr count: 3174
2021/12/14 13:40:33 Waiting for dst bucket item count to get to: 0 curr count: 674
2021/12/14 13:40:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (205.71s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/14 13:40:43 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 13:40:43 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 13:40:43 No eventing-con process running
2021/12/14 13:40:43 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/14 13:40:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:40:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:40:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:40:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:40:56 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:40:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/14 13:41:02 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 13:41:07 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 13:41:12 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 13:41:17 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/14 13:41:17 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/14 13:41:17 Starting bucket ops to source bucket
2021/12/14 13:41:38 Waiting for dst bucket item count to get to: 20000 curr count: 14244
2021/12/14 13:41:43 Waiting for dst bucket item count to get to: 20000 curr count: 14244
2021/12/14 13:41:43 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/14 13:41:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:41:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:41:56 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/14 13:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:42:08 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:42:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:42:08 Starting bucket ops to source bucket
2021/12/14 13:42:21 Waiting for dst bucket item count to get to: 10000 curr count: 992
2021/12/14 13:42:26 Waiting for dst bucket item count to get to: 10000 curr count: 5992
2021/12/14 13:42:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 13:42:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:42:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:42:31 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 13:42:36 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 13:42:41 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 13:42:46 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/14 13:42:46 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/14 13:42:46 No eventing-con process running
2021/12/14 13:42:46 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/14 13:42:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:42:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:42:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:42:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:43:09 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:43:14 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:43:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (161.13s)
=== RUN   TestTimerOverwrite
2021/12/14 13:43:24 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 13:43:24 App: TestTimerOverwrite got un-deployed
2021/12/14 13:43:24 No eventing-con process running
2021/12/14 13:43:24 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/14 13:43:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:43:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:43:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:43:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:43:37 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:43:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/14 13:43:43 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 13:43:48 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 13:43:53 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 13:43:58 Waiting for app: TestTimerOverwrite to get deployed
2021/12/14 13:43:58 App: TestTimerOverwrite got deployed
2021/12/14 13:44:08 Starting bucket ops to source bucket
2021/12/14 13:44:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:44:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:45:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:45:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:45:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 13:45:18 Waiting for dst bucket item count to get to: 5000 curr count: 2493
2021/12/14 13:45:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:45:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:45:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:45:33 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 13:45:38 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 13:45:43 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 13:45:48 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/14 13:45:48 App: TestTimerOverwrite got un-deployed
2021/12/14 13:45:48 No eventing-con process running
2021/12/14 13:45:48 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/14 13:45:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:45:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:46:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 13:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:46:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (181.99s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/14 13:46:21 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 13:46:22 Removing node(s): [] from the cluster
2021/12/14 13:46:27 Rebalance progress: 100
2021/12/14 13:46:27 Failing over: [127.0.0.1:19003]
2021/12/14 13:46:32 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 13:46:32 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 13:46:32 No eventing-con process running
2021/12/14 13:46:32 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 13:46:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:46:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:46:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:46:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:46:45 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:46:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/14 13:46:46 Starting bucket ops to source bucket
2021/12/14 13:46:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:46:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:46:56 Context: Post to set retry counter, response code: 200 dump: 
2021/12/14 13:46:56 set retry counter, response &{[] }
2021/12/14 13:48:01 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 13:48:01 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 13:48:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:48:01 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/14 13:48:06 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/14 13:48:06 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/14 13:48:06 No eventing-con process running
2021/12/14 13:48:06 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/14 13:48:07 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:48:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:48:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:48:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:48:29 Waiting for dst bucket item count to get to: 0 curr count: 1666
2021/12/14 13:48:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:48:29 Removing node(s): [] from the cluster
2021/12/14 13:48:34 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.13s)
=== RUN   TestOnUpdateSrcMutation
2021/12/14 13:48:44 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 13:48:44 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 13:48:44 No eventing-con process running
2021/12/14 13:48:44 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/14 13:48:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:48:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:48:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:48:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:48:57 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:48:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 13:48:59 Starting bucket ops to source bucket
2021/12/14 13:49:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 13:49:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:49:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:49:31 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 13:49:36 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 13:49:41 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 13:49:46 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/14 13:49:46 App: TestOnUpdateSrcMutation got un-deployed
2021/12/14 13:49:46 No eventing-con process running
2021/12/14 13:49:46 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/14 13:49:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:49:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 13:49:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:49:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:50:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.60s)
=== RUN   TestOnDeleteSrcMutation
2021/12/14 13:50:14 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 13:50:14 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 13:50:14 No eventing-con process running
2021/12/14 13:50:14 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/14 13:50:15 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:50:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:50:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:50:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:50:27 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:50:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/14 13:50:28 Starting bucket ops to source bucket
2021/12/14 13:50:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:50:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:50:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:51:02 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 13:51:07 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 13:51:12 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 13:51:17 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/14 13:51:17 App: TestOnDeleteSrcMutation got un-deployed
2021/12/14 13:51:17 No eventing-con process running
2021/12/14 13:51:17 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/14 13:51:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:51:28 Waiting for dst bucket item count to get to: 0 curr count: 4494
2021/12/14 13:51:33 Waiting for dst bucket item count to get to: 0 curr count: 1994
2021/12/14 13:51:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:51:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:51:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.32s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/14 13:51:50 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 13:51:50 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 13:51:50 No eventing-con process running
2021/12/14 13:51:50 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/14 13:51:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:51:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:52:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:52:03 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:52:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 13:52:04 Starting bucket ops to source bucket
2021/12/14 13:52:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 13:52:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:52:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:52:51 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 13:52:56 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 13:53:01 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 13:53:06 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/14 13:53:06 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/14 13:53:06 No eventing-con process running
2021/12/14 13:53:07 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/14 13:53:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:53:18 Waiting for dst bucket item count to get to: 0 curr count: 5885
2021/12/14 13:53:23 Waiting for dst bucket item count to get to: 0 curr count: 885
2021/12/14 13:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:53:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:53:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.54s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/14 13:53:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 13:53:40 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 13:53:40 No eventing-con process running
2021/12/14 13:53:40 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/14 13:53:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:53:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:53:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:53:53 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:53:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/14 13:53:54 Starting bucket ops to source bucket
2021/12/14 13:54:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 13:54:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 13:54:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:54:43 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 13:54:48 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 13:54:53 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 13:54:58 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/14 13:54:58 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/14 13:54:58 No eventing-con process running
2021/12/14 13:54:58 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/14 13:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:55:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 13:55:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 13:55:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:55:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:55:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.43s)
=== RUN   TestInterHandlerRecursion
2021/12/14 13:55:31 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:55:31 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 13:55:31 No eventing-con process running
2021/12/14 13:55:31 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 13:55:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:55:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:55:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:55:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:55:49 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 13:55:49 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 13:55:49 No eventing-con process running
2021/12/14 13:55:49 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 13:55:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:55:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:55:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:56:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:56:02 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:56:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 13:56:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 13:56:09 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 13:56:14 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 13:56:19 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 13:56:24 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 13:56:24 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 13:56:24 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:56:24 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/14 13:56:24 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/14 13:56:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:56:29 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:56:34 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:56:39 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:56:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:56:44 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 13:56:44 No eventing-con process running
2021/12/14 13:56:44 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 13:56:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:56:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:56:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:56:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:57:02 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 13:57:02 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 13:57:02 No eventing-con process running
2021/12/14 13:57:02 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 13:57:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:57:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:57:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.04s)
=== RUN   TestInterBucketRecursion
2021/12/14 13:57:20 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 13:57:21 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 13:57:21 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:57:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 13:57:26 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 13:57:31 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 13:57:36 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 13:57:41 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/14 13:57:41 App: TestInterBucketRecursion_function1 got deployed
2021/12/14 13:57:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:57:46 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 13:57:51 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 13:57:56 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 13:58:01 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/14 13:58:01 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/14 13:58:01 No eventing-con process running
2021/12/14 13:58:01 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 13:58:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:58:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:58:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:58:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:58:19 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/14 13:58:19 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/14 13:58:19 No eventing-con process running
2021/12/14 13:58:19 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/14 13:58:21 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:58:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:58:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.51s)
=== RUN   TestLargeHandler
2021/12/14 13:58:33 Changed force_compress value to false
2021/12/14 13:58:33 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:58:33 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/14 13:58:33 Changed force_compress value to true
2021/12/14 13:58:33 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/14 13:58:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/14 13:58:40 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 13:58:45 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 13:58:50 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 13:58:55 Waiting for app: TestLargeHandler_function to get deployed
2021/12/14 13:58:55 App: TestLargeHandler_function got deployed
2021/12/14 13:58:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 13:59:00 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 13:59:05 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 13:59:10 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/14 13:59:10 App: TestLargeHandler_function got un-deployed
2021/12/14 13:59:10 No eventing-con process running
2021/12/14 13:59:10 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/14 13:59:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:59:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (50.43s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/14 13:59:33 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 13:59:33 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 13:59:33 No eventing-con process running
2021/12/14 13:59:33 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/14 13:59:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:59:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:59:41 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:59:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:59:51 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 13:59:51 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 13:59:51 No eventing-con process running
2021/12/14 13:59:51 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/14 13:59:52 Context: Bucket flush, response code: 200 dump: 
2021/12/14 13:59:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 13:59:59 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:00:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:00:04 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:00:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/14 14:00:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:00:10 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 14:00:15 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 14:00:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 14:00:25 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/14 14:00:25 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/14 14:00:25 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:00:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/14 14:00:27 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:00:32 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 14:00:37 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 14:00:42 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 14:00:47 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/14 14:00:47 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/14 14:00:47 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/14 14:00:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:00:52 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 14:00:57 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 14:01:02 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 14:01:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/14 14:01:07 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/14 14:01:07 No eventing-con process running
2021/12/14 14:01:07 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/14 14:01:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:01:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:01:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:01:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:01:26 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 14:01:31 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 14:01:36 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 14:01:41 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/14 14:01:41 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/14 14:01:41 No eventing-con process running
2021/12/14 14:01:41 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/14 14:01:43 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:01:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:01:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.63s)
=== RUN   TestAllowInterBucketRecursion
2021/12/14 14:02:00 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:02:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 14:02:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:02:06 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 14:02:11 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 14:02:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 14:02:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/14 14:02:21 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/14 14:02:21 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:02:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 14:02:22 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:02:27 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 14:02:32 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 14:02:37 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 14:02:42 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/14 14:02:42 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/14 14:02:42 TestAllowInterBucketRecursion_function2 is deployed
2021/12/14 14:02:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:02:47 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 14:02:52 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 14:02:57 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 14:03:02 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/14 14:03:02 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/14 14:03:02 No eventing-con process running
2021/12/14 14:03:02 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/14 14:03:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:03:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:03:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:03:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:03:21 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 14:03:26 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 14:03:31 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 14:03:36 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/14 14:03:36 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/14 14:03:36 No eventing-con process running
2021/12/14 14:03:36 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/14 14:03:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:03:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:03:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:03:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.08s)
=== RUN   TestInterBucketRecursion2
2021/12/14 14:03:56 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:04:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:04:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:04:14 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:04:19 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:04:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 14:04:25 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 14:04:30 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 14:04:35 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 14:04:40 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/14 14:04:40 App: TestInterBucketRecursion2_function1 got deployed
2021/12/14 14:04:40 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:04:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 14:04:46 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 14:04:51 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 14:04:56 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 14:05:01 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/14 14:05:01 App: TestInterBucketRecursion2_function2 got deployed
2021/12/14 14:05:01 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:05:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 14:05:07 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 14:05:12 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 14:05:17 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 14:05:22 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/14 14:05:22 App: TestInterBucketRecursion2_function3 got deployed
2021/12/14 14:05:22 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:05:23 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/14 14:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:05:28 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 14:05:33 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 14:05:38 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 14:05:43 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/14 14:05:43 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/14 14:05:43 No eventing-con process running
2021/12/14 14:05:43 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/14 14:05:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:05:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:05:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:05:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:06:01 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 14:06:06 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 14:06:11 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 14:06:16 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/14 14:06:16 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/14 14:06:16 No eventing-con process running
2021/12/14 14:06:16 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/14 14:06:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:06:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:06:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:06:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:06:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:06:35 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 14:06:40 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 14:06:45 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 14:06:50 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/14 14:06:50 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/14 14:06:50 No eventing-con process running
2021/12/14 14:06:50 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/14 14:06:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:06:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:06:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:07:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:07:08 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/14 14:07:08 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/14 14:07:08 No eventing-con process running
2021/12/14 14:07:08 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/14 14:07:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:07:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.40s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/14 14:07:28 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 14:07:33 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/14 14:07:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:07:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:08:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:08:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:08:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:08:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:09:06 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:09:21 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:09:36 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/14 14:09:51 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:62} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/14 14:10:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:10:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 14:10:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 14:10:12 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 14:10:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:10:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 14:10:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 14:10:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/14 14:10:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:10:22 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 14:10:22 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 14:10:22 eventingSetup.go:46: No eventing-con process running
2021/12/14 14:10:22 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/14 14:10:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:10:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:10:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:10:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:10:40 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:10:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 14:10:46 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 14:10:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 14:10:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 14:11:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/14 14:11:01 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 55dc35816e34b0d44cca874a1c0a51e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: dc8924a14720ee86963beef25c033c83
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: 4e3cbb387e6b356ac9e7a88d768262d3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/14 14:11:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:11:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 14:11:07 Rebalance progress: 29.16666666666666
2021/12/14 14:11:12 Rebalance progress: 29.16666666666666
2021/12/14 14:11:17 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: dc8924a14720ee86963beef25c033c83
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
Producer node: 127.0.0.1:9303	Node UUID: 4e3cbb387e6b356ac9e7a88d768262d3
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
Producer node: 127.0.0.1:9302	Node UUID: 55dc35816e34b0d44cca874a1c0a51e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/14 14:11:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:11:17 handler_test.go:1674: Deleting source bucket: default
2021/12/14 14:11:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 14:11:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 14:11:24 Rebalance progress: 29.16666666666666
2021/12/14 14:11:29 Rebalance progress: 29.16666666666666
2021/12/14 14:11:34 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 130
	vb list dump: [4, 16, 19, 23, 29, 32, 34, 42, 50, 54, 57, 66, 113, 120, 123, 145, 157, 160, 167, 180, 184, 187, 199, 206, 224, 227, 245, 249, 259, 263, 273-274, 293, 300, 310, 314, 324, 326, 328, 332, 337, 342, 348, 351, 353, 366-367, 372, 376, 392-393, 410, 423, 428, 436, 513, 517, 525, 528, 532, 534, 543, 549, 553, 560, 564, 571, 575, 577, 587-588, 593, 598-599, 615, 622, 631, 634, 640, 646, 648, 658, 665, 667, 677-678, 680, 683, 692-694, 705, 714, 716, 719, 722, 727, 751, 753, 761, 764, 771, 786, 803, 807, 813, 821, 836, 847-848, 855, 861, 864, 873, 906, 908, 910, 912, 916, 922, 924, 932, 955, 957-958, 977, 980, 984, 988, 991]
	stream status: running
	len: 894
	vb list dump: [0-3, 5-15, 17-18, 20-22, 24-28, 30-31, 33, 35-41, 43-49, 51-53, 55-56, 58-65, 67-112, 114-119, 121-122, 124-144, 146-156, 158-159, 161-166, 168-179, 181-183, 185-186, 188-198, 200-205, 207-223, 225-226, 228-244, 246-248, 250-258, 260-262, 264-272, 275-292, 294-299, 301-309, 311-313, 315-323, 325, 327, 329-331, 333-336, 338-341, 343-347, 349-350, 352, 354-365, 368-371, 373-375, 377-391, 394-409, 411-422, 424-427, 429-435, 437-512, 514-516, 518-524, 526-527, 529-531, 533, 535-542, 544-548, 550-552, 554-559, 561-563, 565-570, 572-574, 576, 578-586, 589-592, 594-597, 600-614, 616-621, 623-630, 632-633, 635-639, 641-645, 647, 649-657, 659-664, 666, 668-676, 679, 681-682, 684-691, 695-704, 706-713, 715, 717-718, 720-721, 723-726, 728-750, 752, 754-760, 762-763, 765-770, 772-785, 787-802, 804-806, 808-812, 814-820, 822-835, 837-846, 849-854, 856-860, 862-863, 865-872, 874-905, 907, 909, 911, 913-915, 917-921, 923, 925-931, 933-954, 956, 959-976, 978-979, 981-983, 985-987, 989-990, 992-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 55dc35816e34b0d44cca874a1c0a51e3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-3, 5-15, 17-18, 20-22, 24-28, 30-31, 33, 35-41, 43-49, 51-53, 55-56, 58-65, 67-112]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-119, 121-122, 124-144, 146-156, 158-159, 161-166, 168-179, 181-183, 185-186, 188-198, 200-205, 207-223, 225-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [228-244, 246-248, 250-258, 260-262, 264-272, 275-292, 294-299, 301-309, 311-313, 315-323, 325, 327, 329-331, 333-336, 338-341]
Producer node: 172.31.2.134:9300	Node UUID: dc8924a14720ee86963beef25c033c83
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [911, 913-915, 917-921, 923, 925-931, 933-954, 956, 959-976, 978-979, 981-983, 985-987, 989-990, 992-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 98
	v: [684-691, 695-704, 706-713, 715, 717-718, 720-721, 723-726, 728-750, 752, 754-760, 762-763, 765-770, 772-785, 787-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [797-802, 804-806, 808-812, 814-820, 822-835, 837-846, 849-854, 856-860, 862-863, 865-872, 874-905, 907, 909]
Producer node: 127.0.0.1:9303	Node UUID: 4e3cbb387e6b356ac9e7a88d768262d3
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [343-347, 349-350, 352, 354-365, 368-371, 373-375, 377-391, 394-409, 411-422, 424-427, 429-435, 437-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 103
	v: [456-512, 514-516, 518-524, 526-527, 529-531, 533, 535-542, 544-548, 550-552, 554-559, 561-563, 565-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 92
	v: [570, 572-574, 576, 578-586, 589-592, 594-597, 600-614, 616-621, 623-630, 632-633, 635-639, 641-645, 647, 649-657, 659-664, 666, 668-676, 679, 681-682]


2021/12/14 14:11:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:11:39 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/14 14:11:39 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/14 14:11:44 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/14 14:11:44 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/14 14:11:44 eventingSetup.go:46: No eventing-con process running
2021/12/14 14:11:44 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/14 14:11:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:11:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:11:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:11:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:11:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 14:11:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 14:12:03 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (282.52s)
=== RUN   TestN1QLRecursion
2021/12/14 14:12:06 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 14:12:06 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 14:12:06 Removing node(s): [] from the cluster
2021/12/14 14:12:11 Rebalance progress: 100
2021/12/14 14:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:12:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:12:34 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:12:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 14:12:35 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:12:40 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 14:12:45 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 14:12:50 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 14:12:55 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/14 14:12:55 App: TestN1QLRecursion_function1 got deployed
2021/12/14 14:12:55 TestN1QLRecursion_function1 is deployed
2021/12/14 14:12:55 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:12:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 14:12:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:13:01 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 14:13:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 14:13:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 14:13:16 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/14 14:13:16 App: TestN1QLRecursion_function3 got deployed
2021/12/14 14:13:16 TestN1QLRecursion_function3 is deployed
2021/12/14 14:13:16 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:13:16 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/14 14:13:16 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/14 14:13:16 Success: TestN1QLRecursion_function2 is not deployed
2021/12/14 14:13:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:13:21 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 14:13:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 14:13:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 14:13:37 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/14 14:13:37 App: TestN1QLRecursion_function1 got un-deployed
2021/12/14 14:13:37 No eventing-con process running
2021/12/14 14:13:37 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/14 14:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:13:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:13:45 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:13:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:13:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:13:55 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 14:14:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 14:14:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 14:14:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/14 14:14:10 App: TestN1QLRecursion_function3 got un-deployed
2021/12/14 14:14:10 No eventing-con process running
2021/12/14 14:14:10 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/14 14:14:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:14:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:14:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:14:26 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 14:14:31 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.06s)
=== RUN   TestN1QLAllowRecursion
2021/12/14 14:14:31 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 14:14:31 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 14:14:31 Removing node(s): [] from the cluster
2021/12/14 14:14:36 Rebalance progress: 100
2021/12/14 14:14:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:14:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:14:59 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:15:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/14 14:15:00 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:15:05 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 14:15:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 14:15:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 14:15:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/14 14:15:20 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/14 14:15:20 TestN1QLAllowRecursion_function1 is deployed
2021/12/14 14:15:20 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:15:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/14 14:15:21 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:15:26 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 14:15:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 14:15:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 14:15:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/14 14:15:41 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/14 14:15:41 TestN1QLAllowRecursion_function3 is deployed
2021/12/14 14:15:41 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:15:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/14 14:15:42 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/14 14:15:47 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 14:15:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 14:15:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 14:16:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/14 14:16:02 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/14 14:16:02 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/14 14:16:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:16:07 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 14:16:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 14:16:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 14:16:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/14 14:16:22 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/14 14:16:22 No eventing-con process running
2021/12/14 14:16:22 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/14 14:16:24 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:16:31 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:16:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:16:41 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 14:16:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 14:16:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 14:16:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/14 14:16:56 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/14 14:16:56 No eventing-con process running
2021/12/14 14:16:56 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/14 14:16:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:17:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:17:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:17:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:17:15 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 14:17:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 14:17:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 14:17:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/14 14:17:30 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/14 14:17:30 No eventing-con process running
2021/12/14 14:17:30 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/14 14:17:32 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:17:39 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:17:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:17:46 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 14:17:51 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.94s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/14 14:17:51 Sleeping for some time
2021/12/14 14:18:01 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 14:18:01 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 14:18:01 No eventing-con process running
2021/12/14 14:18:01 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/14 14:18:02 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:18:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:18:09 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:18:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:18:14 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:18:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 14:18:15 Starting bucket ops to source bucket
2021/12/14 14:19:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 14:19:10 Waiting for dst bucket item count to get to: 5000 curr count: 795
2021/12/14 14:19:15 Waiting for dst bucket item count to get to: 5000 curr count: 795
2021/12/14 14:19:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 14:19:15 Starting bucket ops to source bucket
2021/12/14 14:19:16 Starting bucket ops to source bucket
2021/12/14 14:20:22 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/14 14:20:22 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/14 14:20:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:20:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:20:27 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 14:20:32 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 14:20:37 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 14:20:42 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/14 14:20:42 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/14 14:20:42 No eventing-con process running
2021/12/14 14:20:42 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/14 14:20:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:20:54 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:20:59 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/14 14:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:21:10 Waiting for dst bucket item count to get to: 0 curr count: 5861
2021/12/14 14:21:15 Waiting for dst bucket item count to get to: 0 curr count: 5861
2021/12/14 14:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.91s)
=== RUN   TestTimerOverWriteSameReference
2021/12/14 14:21:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 14:21:21 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 14:21:21 No eventing-con process running
2021/12/14 14:21:21 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/14 14:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:21:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:21:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:21:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:21:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/14 14:21:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 14:21:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 14:21:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 14:21:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/14 14:21:55 App: TestTimerOverWriteSameReference got deployed
2021/12/14 14:22:05 Starting bucket ops to source bucket
2021/12/14 14:23:07 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/14 14:23:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:23:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:23:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 14:23:27 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 14:23:32 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 14:23:37 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/14 14:23:37 App: TestTimerOverWriteSameReference got un-deployed
2021/12/14 14:23:37 No eventing-con process running
2021/12/14 14:23:37 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/14 14:23:38 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:23:48 Waiting for dst bucket item count to get to: 0 curr count: 854
2021/12/14 14:23:53 Waiting for dst bucket item count to get to: 0 curr count: 354
2021/12/14 14:23:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:23:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:24:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/14 14:24:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.80s)
=== RUN   TestCancelTimerBucketop
2021/12/14 14:24:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 14:24:10 App: TestCancelTimerBucketop got un-deployed
2021/12/14 14:24:10 No eventing-con process running
2021/12/14 14:24:10 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/14 14:24:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:24:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:24:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:24:24 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:24:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/14 14:24:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 14:24:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 14:24:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 14:24:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/14 14:24:45 App: TestCancelTimerBucketop got deployed
2021/12/14 14:24:45 Starting bucket ops to source bucket
2021/12/14 14:24:53 Starting bucket ops to source bucket
2021/12/14 14:25:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 14:25:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:25:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:26:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 14:26:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 14:26:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 14:26:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/14 14:26:15 App: TestCancelTimerBucketop got un-deployed
2021/12/14 14:26:15 No eventing-con process running
2021/12/14 14:26:15 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/14 14:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:26:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:26:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:26:39 Waiting for dst bucket item count to get to: 0 curr count: 957
2021/12/14 14:26:44 Waiting for dst bucket item count to get to: 0 curr count: 457
2021/12/14 14:26:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.53s)
=== RUN   TestJSExpiryDate
2021/12/14 14:26:49 Starting bucket ops to source bucket
2021/12/14 14:27:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 14:27:07 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/14 14:27:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/14 14:27:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:27:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:27:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:27:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 14:27:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 14:27:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 14:27:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/14 14:27:53 App: TestJSExpiryDate got un-deployed
2021/12/14 14:27:53 No eventing-con process running
2021/12/14 14:27:53 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/14 14:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:28:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:28:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.70s)
=== RUN   TestBinaryDoc
2021/12/14 14:28:13 Starting bucket ops to source bucket
2021/12/14 14:28:13 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:28:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/14 14:28:14 Starting bucket ops to source bucket
2021/12/14 14:28:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:28:14 Deleting source bucket docs now
2021/12/14 14:28:14 Starting bucket ops to source bucket
2021/12/14 14:28:51 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/14 14:28:56 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/14 14:28:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/14 14:28:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:28:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:29:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 14:29:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 14:29:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 14:29:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/14 14:29:16 App: TestBinaryDoc got un-deployed
2021/12/14 14:29:16 No eventing-con process running
2021/12/14 14:29:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/14 14:29:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:29:28 Waiting for dst bucket item count to get to: 0 curr count: 22
2021/12/14 14:29:33 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/14 14:29:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:29:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:29:45 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/14 14:29:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.33s)
=== RUN   TestConstantBindings
2021/12/14 14:29:45 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:29:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/14 14:29:46 Starting bucket ops to source bucket
2021/12/14 14:29:52 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 14:29:57 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 14:30:02 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/14 14:30:07 Waiting for dst bucket item count to get to: 6000 curr count: 2186
2021/12/14 14:30:12 Waiting for dst bucket item count to get to: 6000 curr count: 2186
2021/12/14 14:30:12 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/14 14:30:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:30:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:30:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 14:30:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 14:30:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 14:30:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/14 14:30:32 App: TestConstantBindings got un-deployed
2021/12/14 14:30:32 No eventing-con process running
2021/12/14 14:30:32 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/14 14:30:34 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:30:44 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:30:49 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:31:01 Waiting for dst bucket item count to get to: 0 curr count: 5237
2021/12/14 14:31:06 Waiting for dst bucket item count to get to: 0 curr count: 5237
2021/12/14 14:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.75s)
=== RUN   TestCodeInsight
2021/12/14 14:31:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 14:31:11 App: TestCodeInsight got un-deployed
2021/12/14 14:31:11 No eventing-con process running
2021/12/14 14:31:11 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/14 14:31:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:31:19 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:31:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:31:24 Starting bucket ops to source bucket
2021/12/14 14:31:25 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:31:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/14 14:31:31 Waiting for app: TestCodeInsight to get deployed
2021/12/14 14:31:36 Waiting for app: TestCodeInsight to get deployed
2021/12/14 14:31:41 Waiting for app: TestCodeInsight to get deployed
2021/12/14 14:31:46 Waiting for app: TestCodeInsight to get deployed
2021/12/14 14:31:46 App: TestCodeInsight got deployed
2021/12/14 14:31:51 Waiting for dst bucket item count to get to: 1000 curr count: 477
2021/12/14 14:31:51 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 14:31:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:31:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 14:32:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 14:32:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 14:32:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/14 14:32:11 App: TestCodeInsight got un-deployed
2021/12/14 14:32:11 No eventing-con process running
2021/12/14 14:32:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/14 14:32:13 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:32:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:32:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:32:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:32:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:32:40 Waiting for dst bucket item count to get to: 0 curr count: 662
2021/12/14 14:32:45 Waiting for dst bucket item count to get to: 0 curr count: 662
2021/12/14 14:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.59s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13004.184s

Functional tests took 3 hours, 36 minutes and 48 seconds.

N1ql tests

2021/12/14 14:32:54 Started n1ql
2021/12/14 14:33:04 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/14 14:33:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/14 14:33:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 14:33:04 Removing node(s): [] from the cluster
2021/12/14 14:33:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/14 14:33:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/14 14:33:14 App: TestRecursiveMutationN1QL got un-deployed
2021/12/14 14:33:14 No eventing-con process running
2021/12/14 14:33:14 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/14 14:33:16 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:33:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:33:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:33:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:33:28 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:33:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.68s)
=== RUN   TestFlexReset1
2021/12/14 14:33:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 14:33:33 App: TestFlexReset1 got un-deployed
2021/12/14 14:33:33 No eventing-con process running
2021/12/14 14:33:33 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/14 14:33:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:33:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:33:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:33:47 Starting bucket ops to source bucket
2021/12/14 14:33:47 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:33:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/14 14:33:54 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 14:33:59 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 14:34:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 14:34:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/14 14:34:09 App: TestFlexReset1 got deployed
2021/12/14 14:34:14 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/14 14:34:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 14:34:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:34:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:34:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 14:34:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 14:34:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 14:34:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/14 14:34:34 App: TestFlexReset1 got un-deployed
2021/12/14 14:34:34 No eventing-con process running
2021/12/14 14:34:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/14 14:34:36 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:34:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:34:57 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.24s)
=== RUN   TestFlexReset2
2021/12/14 14:35:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 14:35:02 App: TestFlexReset2 got un-deployed
2021/12/14 14:35:02 No eventing-con process running
2021/12/14 14:35:02 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/14 14:35:04 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:35:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:35:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:35:16 Starting bucket ops to source bucket
2021/12/14 14:35:17 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:35:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/14 14:35:23 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 14:35:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 14:35:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 14:35:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/14 14:35:38 App: TestFlexReset2 got deployed
2021/12/14 14:35:43 Waiting for dst bucket item count to get to: 1000 curr count: 16
2021/12/14 14:35:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/14 14:35:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:35:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:35:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 14:35:53 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 14:35:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 14:36:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/14 14:36:03 App: TestFlexReset2 got un-deployed
2021/12/14 14:36:03 No eventing-con process running
2021/12/14 14:36:03 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/14 14:36:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:36:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:36:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:36:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/14 14:36:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.28s)
=== RUN   TestN1QLLabelledBreak
2021/12/14 14:36:32 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 14:36:32 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 14:36:32 No eventing-con process running
2021/12/14 14:36:32 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/14 14:36:33 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:36:40 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:36:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:36:45 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:36:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 14:36:51 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 14:36:56 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 14:37:01 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 14:37:06 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/14 14:37:06 App: TestN1QLLabelledBreak got deployed
2021/12/14 14:37:06 Starting bucket ops to source bucket
2021/12/14 14:37:16 Waiting for dst bucket item count to get to: 10000 curr count: 30
2021/12/14 14:37:21 Waiting for dst bucket item count to get to: 10000 curr count: 223
2021/12/14 14:37:26 Waiting for dst bucket item count to get to: 10000 curr count: 223
2021/12/14 14:37:31 Waiting for dst bucket item count to get to: 10000 curr count: 3582
2021/12/14 14:37:36 Waiting for dst bucket item count to get to: 10000 curr count: 3582
2021/12/14 14:37:41 Waiting for dst bucket item count to get to: 10000 curr count: 7633
2021/12/14 14:37:46 Waiting for dst bucket item count to get to: 10000 curr count: 7633
2021/12/14 14:37:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 14:37:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:37:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:37:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 14:37:56 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 14:38:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 14:38:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/14 14:38:06 App: TestN1QLLabelledBreak got un-deployed
2021/12/14 14:38:06 No eventing-con process running
2021/12/14 14:38:06 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/14 14:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:38:18 Waiting for dst bucket item count to get to: 0 curr count: 3561
2021/12/14 14:38:23 Waiting for dst bucket item count to get to: 0 curr count: 1061
2021/12/14 14:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:38:25 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:38:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 14:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.18s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/14 14:38:40 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 14:38:40 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 14:38:40 No eventing-con process running
2021/12/14 14:38:40 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/14 14:38:42 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:38:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:38:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:38:53 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:38:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/14 14:39:00 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 14:39:05 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 14:39:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 14:39:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/14 14:39:15 App: TestN1QLUnlabelledBreak got deployed
2021/12/14 14:39:15 Starting bucket ops to source bucket
2021/12/14 14:39:24 Waiting for dst bucket item count to get to: 10000 curr count: 250
2021/12/14 14:39:29 Waiting for dst bucket item count to get to: 10000 curr count: 746
2021/12/14 14:39:34 Waiting for dst bucket item count to get to: 10000 curr count: 746
2021/12/14 14:39:39 Waiting for dst bucket item count to get to: 10000 curr count: 3270
2021/12/14 14:39:44 Waiting for dst bucket item count to get to: 10000 curr count: 6170
2021/12/14 14:39:49 Waiting for dst bucket item count to get to: 10000 curr count: 8396
2021/12/14 14:39:49 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 14:39:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:39:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:39:54 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 14:39:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 14:40:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 14:40:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/14 14:40:09 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/14 14:40:09 No eventing-con process running
2021/12/14 14:40:09 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/14 14:40:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:40:21 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:40:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:40:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:40:28 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:40:38 Waiting for dst bucket item count to get to: 0 curr count: 7352
2021/12/14 14:40:43 Waiting for dst bucket item count to get to: 0 curr count: 2352
2021/12/14 14:40:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.28s)
=== RUN   TestN1QLThrowStatement
2021/12/14 14:40:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 14:40:48 App: TestN1QLThrowStatement got un-deployed
2021/12/14 14:40:48 No eventing-con process running
2021/12/14 14:40:48 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/14 14:40:50 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:40:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:41:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:41:02 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:41:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/14 14:41:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 14:41:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 14:41:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 14:41:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/14 14:41:23 App: TestN1QLThrowStatement got deployed
2021/12/14 14:41:23 Starting bucket ops to source bucket
2021/12/14 14:41:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 14:41:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/14 14:41:42 Waiting for dst bucket item count to get to: 10000 curr count: 986
2021/12/14 14:41:47 Waiting for dst bucket item count to get to: 10000 curr count: 2959
2021/12/14 14:41:52 Waiting for dst bucket item count to get to: 10000 curr count: 5014
2021/12/14 14:41:57 Waiting for dst bucket item count to get to: 10000 curr count: 7013
2021/12/14 14:42:02 Waiting for dst bucket item count to get to: 10000 curr count: 8984
2021/12/14 14:42:07 Waiting for dst bucket item count to get to: 10000 curr count: 9697
2021/12/14 14:42:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/14 14:42:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:42:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:42:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 14:42:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 14:42:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 14:42:28 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/14 14:42:28 App: TestN1QLThrowStatement got un-deployed
2021/12/14 14:42:28 No eventing-con process running
2021/12/14 14:42:28 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/14 14:42:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:42:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 14:42:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 14:42:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:42:46 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:42:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/14 14:42:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.23s)
=== RUN   TestN1QLNestedForLoop
2021/12/14 14:43:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 14:43:01 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 14:43:01 No eventing-con process running
2021/12/14 14:43:01 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/14 14:43:03 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:43:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:43:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:43:15 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:43:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/14 14:43:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 14:43:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 14:43:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 14:43:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/14 14:43:36 App: TestN1QLNestedForLoop got deployed
2021/12/14 14:43:36 Starting bucket ops to source bucket
2021/12/14 14:43:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/14 14:43:50 Waiting for dst bucket item count to get to: 5000 curr count: 147
2021/12/14 14:43:55 Waiting for dst bucket item count to get to: 5000 curr count: 147
2021/12/14 14:44:00 Waiting for dst bucket item count to get to: 5000 curr count: 2216
2021/12/14 14:44:05 Waiting for dst bucket item count to get to: 5000 curr count: 2216
2021/12/14 14:44:10 Waiting for dst bucket item count to get to: 5000 curr count: 4611
2021/12/14 14:44:15 Waiting for dst bucket item count to get to: 5000 curr count: 4611
2021/12/14 14:44:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 14:44:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:44:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:44:20 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 14:44:26 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 14:44:31 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 14:44:36 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/14 14:44:36 App: TestN1QLNestedForLoop got un-deployed
2021/12/14 14:44:36 No eventing-con process running
2021/12/14 14:44:36 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/14 14:44:37 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:44:47 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 14:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:44:49 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:44:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:45:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/14 14:45:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (127.91s)
=== RUN   TestN1QLAutoClose
2021/12/14 14:45:09 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 14:45:09 App: TestN1QLAutoClose got un-deployed
2021/12/14 14:45:09 No eventing-con process running
2021/12/14 14:45:09 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/14 14:45:11 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:45:23 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:45:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/14 14:45:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 14:45:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 14:45:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 14:45:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/14 14:45:44 App: TestN1QLAutoClose got deployed
2021/12/14 14:45:44 Starting bucket ops to source bucket
2021/12/14 14:45:50 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/14 14:45:55 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/14 14:46:00 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/14 14:46:05 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/14 14:46:10 Waiting for dst bucket item count to get to: 500 curr count: 29
2021/12/14 14:46:15 Waiting for dst bucket item count to get to: 500 curr count: 39
2021/12/14 14:46:20 Waiting for dst bucket item count to get to: 500 curr count: 49
2021/12/14 14:46:25 Waiting for dst bucket item count to get to: 500 curr count: 60
2021/12/14 14:46:30 Waiting for dst bucket item count to get to: 500 curr count: 73
2021/12/14 14:46:35 Waiting for dst bucket item count to get to: 500 curr count: 85
2021/12/14 14:46:40 Waiting for dst bucket item count to get to: 500 curr count: 95
2021/12/14 14:46:45 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/14 14:46:50 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/14 14:46:55 Waiting for dst bucket item count to get to: 500 curr count: 126
2021/12/14 14:47:00 Waiting for dst bucket item count to get to: 500 curr count: 136
2021/12/14 14:47:05 Waiting for dst bucket item count to get to: 500 curr count: 136
2021/12/14 14:47:10 Waiting for dst bucket item count to get to: 500 curr count: 158
2021/12/14 14:47:15 Waiting for dst bucket item count to get to: 500 curr count: 158
2021/12/14 14:47:20 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/14 14:47:25 Waiting for dst bucket item count to get to: 500 curr count: 182
2021/12/14 14:47:30 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/14 14:47:35 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/14 14:47:40 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/14 14:47:45 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/14 14:47:50 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 14:47:55 Waiting for dst bucket item count to get to: 500 curr count: 244
2021/12/14 14:48:00 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/14 14:48:05 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/14 14:48:10 Waiting for dst bucket item count to get to: 500 curr count: 286
2021/12/14 14:48:15 Waiting for dst bucket item count to get to: 500 curr count: 286
2021/12/14 14:48:20 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/14 14:48:25 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/14 14:48:30 Waiting for dst bucket item count to get to: 500 curr count: 330
2021/12/14 14:48:35 Waiting for dst bucket item count to get to: 500 curr count: 330
2021/12/14 14:48:40 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/14 14:48:45 Waiting for dst bucket item count to get to: 500 curr count: 350
2021/12/14 14:48:50 Waiting for dst bucket item count to get to: 500 curr count: 371
2021/12/14 14:48:55 Waiting for dst bucket item count to get to: 500 curr count: 371
2021/12/14 14:49:00 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/14 14:49:05 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/14 14:49:10 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/14 14:49:15 Waiting for dst bucket item count to get to: 500 curr count: 426
2021/12/14 14:49:20 Waiting for dst bucket item count to get to: 500 curr count: 435
2021/12/14 14:49:25 Waiting for dst bucket item count to get to: 500 curr count: 446
2021/12/14 14:49:30 Waiting for dst bucket item count to get to: 500 curr count: 456
2021/12/14 14:49:35 Waiting for dst bucket item count to get to: 500 curr count: 469
2021/12/14 14:49:40 Waiting for dst bucket item count to get to: 500 curr count: 480
2021/12/14 14:49:45 Waiting for dst bucket item count to get to: 500 curr count: 488
2021/12/14 14:49:50 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/14 14:49:50 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/14 14:49:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:49:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:49:55 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 14:50:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 14:50:05 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 14:50:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/14 14:50:10 App: TestN1QLAutoClose got un-deployed
2021/12/14 14:50:10 No eventing-con process running
2021/12/14 14:50:10 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/14 14:50:12 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:50:22 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:50:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:50:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:50:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:50:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/14 14:50:44 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/14 14:50:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.72s)
=== RUN   TestN1QLPosParams
2021/12/14 14:50:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 14:50:49 App: TestN1QLPosParams got un-deployed
2021/12/14 14:50:49 No eventing-con process running
2021/12/14 14:50:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/14 14:50:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:50:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:50:58 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:51:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:51:03 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:51:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/14 14:51:09 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 14:51:14 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 14:51:19 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 14:51:24 Waiting for app: TestN1QLPosParams to get deployed
2021/12/14 14:51:24 App: TestN1QLPosParams got deployed
2021/12/14 14:51:24 Starting bucket ops to source bucket
2021/12/14 14:51:33 Waiting for dst bucket item count to get to: 5000 curr count: 551
2021/12/14 14:51:38 Waiting for dst bucket item count to get to: 5000 curr count: 1655
2021/12/14 14:51:43 Waiting for dst bucket item count to get to: 5000 curr count: 1655
2021/12/14 14:51:48 Waiting for dst bucket item count to get to: 5000 curr count: 3604
2021/12/14 14:51:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 14:51:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:51:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:51:53 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 14:51:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 14:52:04 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 14:52:09 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/14 14:52:09 App: TestN1QLPosParams got un-deployed
2021/12/14 14:52:09 No eventing-con process running
2021/12/14 14:52:09 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/14 14:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:52:21 Waiting for dst bucket item count to get to: 0 curr count: 4161
2021/12/14 14:52:26 Waiting for dst bucket item count to get to: 0 curr count: 4161
2021/12/14 14:52:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:52:27 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:52:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:52:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (113.37s)
=== RUN   TestN1QLExhaustConnPool
2021/12/14 14:52:42 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 14:52:42 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 14:52:42 No eventing-con process running
2021/12/14 14:52:42 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/14 14:52:44 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:52:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:52:51 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:52:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:52:56 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:52:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/14 14:53:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 14:53:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 14:53:12 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 14:53:17 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/14 14:53:17 App: TestN1QLExhaustConnPool got deployed
2021/12/14 14:53:17 Starting bucket ops to source bucket
2021/12/14 14:53:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/14 14:53:27 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/14 14:53:32 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/14 14:53:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/14 14:53:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:53:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:53:38 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 14:53:43 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 14:53:48 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 14:53:53 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/14 14:53:53 App: TestN1QLExhaustConnPool got un-deployed
2021/12/14 14:53:53 No eventing-con process running
2021/12/14 14:53:53 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/14 14:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:54:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 14:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:54:06 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:54:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/14 14:54:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.91s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/14 14:54:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 14:54:21 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 14:54:21 No eventing-con process running
2021/12/14 14:54:21 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/14 14:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:54:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:54:30 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:54:35 Starting bucket ops to source bucket
2021/12/14 14:54:52 Starting bucket ops to source bucket
2021/12/14 14:55:10 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 9890
2021/12/14 14:55:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/14 14:55:16 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 14:55:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 14:55:26 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 14:55:31 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/14 14:55:31 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/14 14:55:31 Starting bucket ops to source bucket
2021/12/14 14:55:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:55:41 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:55:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:55:51 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:55:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:21 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/14 14:56:26 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/14 14:56:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:56:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:56:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 14:56:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 14:56:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 14:56:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/14 14:56:47 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/14 14:56:47 No eventing-con process running
2021/12/14 14:56:47 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/14 14:56:48 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:56:58 Waiting for dst bucket item count to get to: 0 curr count: 24463
2021/12/14 14:57:03 Waiting for dst bucket item count to get to: 0 curr count: 9463
2021/12/14 14:57:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:57:05 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:57:15 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/14 14:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.00s)
=== RUN   TestN1QLQueryConsistency
2021/12/14 14:57:20 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 14:57:20 App: n1qlquery_consistency got un-deployed
2021/12/14 14:57:20 No eventing-con process running
2021/12/14 14:57:20 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/14 14:57:22 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:57:34 Starting bucket ops to source bucket
2021/12/14 14:57:37 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/14 14:57:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/14 14:57:43 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 14:57:48 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 14:57:53 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 14:57:58 Waiting for app: n1qlquery_consistency to get deployed
2021/12/14 14:57:58 App: n1qlquery_consistency got deployed
2021/12/14 14:58:03 Waiting for dst bucket item count to get to: 5000 curr count: 20
2021/12/14 14:58:08 Waiting for dst bucket item count to get to: 5000 curr count: 4708
2021/12/14 14:58:13 Waiting for dst bucket item count to get to: 5000 curr count: 4708
2021/12/14 14:58:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/14 14:58:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/14 14:58:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 14:58:18 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 14:58:23 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 14:58:28 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 14:58:33 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/14 14:58:33 App: n1qlquery_consistency got un-deployed
2021/12/14 14:58:33 No eventing-con process running
2021/12/14 14:58:33 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/14 14:58:35 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:58:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 14:58:47 Context: Bucket flush, response code: 200 dump: 
2021/12/14 14:58:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/14 14:58:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.70s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1558.681s

N1ql tests took 0 hours, 26 minutes and 3 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/14 15:00:32 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/14 15:00:50 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/14 15:00:55 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 15:00:55 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 15:00:55 No eventing-con process running
2021/12/14 15:00:55 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/14 15:00:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:01:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:01:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:01:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:01:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:01:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:01:13 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:01:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 15:01:19 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 15:01:24 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 15:01:29 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 15:01:34 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/14 15:01:34 App: TestDuplicateWithEventingRebIn got deployed
2021/12/14 15:01:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:01:34 Starting bucket ops to source bucket
2021/12/14 15:01:49 Removing node(s): [] from the cluster
2021/12/14 15:01:54 Rebalance progress: 0
2021/12/14 15:01:59 Rebalance progress: 0
2021/12/14 15:02:04 Rebalance progress: 25
2021/12/14 15:02:09 Rebalance progress: 25
2021/12/14 15:02:14 Rebalance progress: 25
2021/12/14 15:02:19 Rebalance progress: 25
2021/12/14 15:02:24 Rebalance progress: 25
2021/12/14 15:02:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:02:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:02:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:02:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 15:02:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:02:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:02:34 Rebalance progress: 25
2021/12/14 15:02:39 Rebalance progress: 25
2021/12/14 15:02:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:02:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 15:02:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:02:44 Waiting for src bucket item count to get to: 200000 curr count: 103931
2021/12/14 15:02:49 Waiting for src bucket item count to get to: 200000 curr count: 103931
2021/12/14 15:02:54 Waiting for src bucket item count to get to: 200000 curr count: 120218
2021/12/14 15:02:59 Waiting for src bucket item count to get to: 200000 curr count: 120218
2021/12/14 15:03:04 Waiting for src bucket item count to get to: 200000 curr count: 136763
2021/12/14 15:03:09 Waiting for src bucket item count to get to: 200000 curr count: 136763
2021/12/14 15:03:14 Waiting for src bucket item count to get to: 200000 curr count: 153133
2021/12/14 15:03:19 Waiting for src bucket item count to get to: 200000 curr count: 153133
2021/12/14 15:03:24 Waiting for src bucket item count to get to: 200000 curr count: 169219
2021/12/14 15:03:29 Waiting for src bucket item count to get to: 200000 curr count: 169219
2021/12/14 15:03:34 Waiting for src bucket item count to get to: 200000 curr count: 185528
2021/12/14 15:03:39 Waiting for src bucket item count to get to: 200000 curr count: 185528
2021/12/14 15:03:49 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:03:54 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:03:59 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:04 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:09 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:14 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:19 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:24 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:29 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:34 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/14 15:04:34 Post SET ops. Expected item count: 200000 got 200001
2021/12/14 15:04:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:04:39 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 15:04:44 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 15:04:49 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 15:04:54 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/14 15:04:54 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/14 15:04:54 No eventing-con process running
2021/12/14 15:04:54 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/14 15:04:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:05:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:05:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:05:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:05:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:05:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:05:18 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/14 15:05:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (277.80s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/14 15:05:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:05:23 Removing node(s): [] from the cluster
2021/12/14 15:05:28 Rebalance progress: 100
2021/12/14 15:05:33 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 15:05:33 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 15:05:33 No eventing-con process running
2021/12/14 15:05:33 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/14 15:05:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:05:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:05:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:05:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:05:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:05:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:05:51 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:05:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 15:05:58 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 15:06:03 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 15:06:08 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 15:06:13 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/14 15:06:13 App: TestDuplicateWithEventingRebOut got deployed
2021/12/14 15:06:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:06:13 Starting bucket ops to source bucket
2021/12/14 15:06:18 Rebalance progress: 25
2021/12/14 15:06:23 Rebalance progress: 25
2021/12/14 15:06:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:06:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 15:06:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:06:28 Waiting for src bucket item count to get to: 200000 curr count: 10517
2021/12/14 15:06:33 Waiting for src bucket item count to get to: 200000 curr count: 26658
2021/12/14 15:06:38 Waiting for src bucket item count to get to: 200000 curr count: 26658
2021/12/14 15:06:43 Waiting for src bucket item count to get to: 200000 curr count: 42999
2021/12/14 15:06:48 Waiting for src bucket item count to get to: 200000 curr count: 42999
2021/12/14 15:06:53 Waiting for src bucket item count to get to: 200000 curr count: 59479
2021/12/14 15:06:58 Waiting for src bucket item count to get to: 200000 curr count: 59479
2021/12/14 15:07:03 Waiting for src bucket item count to get to: 200000 curr count: 59479
2021/12/14 15:07:08 Waiting for src bucket item count to get to: 200000 curr count: 75583
2021/12/14 15:07:13 Waiting for src bucket item count to get to: 200000 curr count: 75583
2021/12/14 15:07:18 Waiting for src bucket item count to get to: 200000 curr count: 91803
2021/12/14 15:07:23 Waiting for src bucket item count to get to: 200000 curr count: 91803
2021/12/14 15:07:28 Waiting for src bucket item count to get to: 200000 curr count: 108425
2021/12/14 15:07:33 Waiting for src bucket item count to get to: 200000 curr count: 108425
2021/12/14 15:07:38 Waiting for src bucket item count to get to: 200000 curr count: 125134
2021/12/14 15:07:43 Waiting for src bucket item count to get to: 200000 curr count: 125134
2021/12/14 15:07:48 Waiting for src bucket item count to get to: 200000 curr count: 141722
2021/12/14 15:07:53 Waiting for src bucket item count to get to: 200000 curr count: 141722
2021/12/14 15:07:58 Waiting for src bucket item count to get to: 200000 curr count: 157943
2021/12/14 15:08:03 Waiting for src bucket item count to get to: 200000 curr count: 157943
2021/12/14 15:08:08 Waiting for src bucket item count to get to: 200000 curr count: 173885
2021/12/14 15:08:13 Waiting for src bucket item count to get to: 200000 curr count: 173885
2021/12/14 15:08:18 Waiting for src bucket item count to get to: 200000 curr count: 189748
2021/12/14 15:08:23 Waiting for src bucket item count to get to: 200000 curr count: 189748
2021/12/14 15:08:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 15:08:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 15:08:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:08:33 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 15:08:38 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 15:08:43 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 15:08:48 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/14 15:08:48 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/14 15:08:48 No eventing-con process running
2021/12/14 15:08:48 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/14 15:08:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:08:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:08:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:08:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:09:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:09:06 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:09:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.81s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/14 15:09:16 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 15:09:16 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 15:09:16 No eventing-con process running
2021/12/14 15:09:16 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/14 15:09:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:09:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:09:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:09:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:09:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:09:35 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:09:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 15:09:41 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 15:09:46 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 15:09:51 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 15:09:56 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/14 15:09:56 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/14 15:09:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:09:56 Starting bucket ops to source bucket
2021/12/14 15:09:56 Removing node(s): [] from the cluster
2021/12/14 15:10:01 Rebalance progress: 25
2021/12/14 15:10:06 Rebalance progress: 25
2021/12/14 15:10:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:10:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/14 15:10:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:10:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:10:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:10:23 Rebalance progress: 16.66666666666667
2021/12/14 15:10:28 Rebalance progress: 16.66666666666667
2021/12/14 15:10:33 Rebalance progress: 16.66666666666667
2021/12/14 15:10:38 Rebalance progress: 37.77412280701754
2021/12/14 15:10:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:10:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:10:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:10:48 Rebalance progress: 16.66666666666667
2021/12/14 15:10:53 Rebalance progress: 16.66666666666667
2021/12/14 15:10:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:10:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:10:58 Waiting for src bucket item count to get to: 200000 curr count: 78187
2021/12/14 15:11:03 Waiting for src bucket item count to get to: 200000 curr count: 93903
2021/12/14 15:11:08 Waiting for src bucket item count to get to: 200000 curr count: 93903
2021/12/14 15:11:13 Waiting for src bucket item count to get to: 200000 curr count: 109834
2021/12/14 15:11:18 Waiting for src bucket item count to get to: 200000 curr count: 109834
2021/12/14 15:11:23 Waiting for src bucket item count to get to: 200000 curr count: 125984
2021/12/14 15:11:28 Waiting for src bucket item count to get to: 200000 curr count: 125984
2021/12/14 15:11:33 Waiting for src bucket item count to get to: 200000 curr count: 141976
2021/12/14 15:11:38 Waiting for src bucket item count to get to: 200000 curr count: 141976
2021/12/14 15:11:43 Waiting for src bucket item count to get to: 200000 curr count: 157901
2021/12/14 15:11:48 Waiting for src bucket item count to get to: 200000 curr count: 157901
2021/12/14 15:11:53 Waiting for src bucket item count to get to: 200000 curr count: 173171
2021/12/14 15:11:58 Waiting for src bucket item count to get to: 200000 curr count: 173171
2021/12/14 15:12:03 Waiting for src bucket item count to get to: 200000 curr count: 189261
2021/12/14 15:12:08 Waiting for src bucket item count to get to: 200000 curr count: 189261
2021/12/14 15:12:13 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 15:12:13 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 15:12:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:12:18 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 15:12:23 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 15:12:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 15:12:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/14 15:12:34 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/14 15:12:34 No eventing-con process running
2021/12/14 15:12:34 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/14 15:12:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:12:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:12:45 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:12:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:12:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:12:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.65s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/14 15:13:07 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 15:13:07 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 15:13:07 No eventing-con process running
2021/12/14 15:13:07 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/14 15:13:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:13:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:13:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:13:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:13:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:13:26 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 15:13:32 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 15:13:37 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 15:13:42 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 15:13:47 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/14 15:13:47 App: TestDuplicateWithKVRebIn got deployed
2021/12/14 15:13:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 15:13:47 Starting bucket ops to source bucket
2021/12/14 15:13:47 Removing node(s): [] from the cluster
2021/12/14 15:13:52 Rebalance progress: 16.66666666666667
2021/12/14 15:13:57 Rebalance progress: 16.66666666666667
2021/12/14 15:14:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:14:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:14:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:14:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:14:07 Rebalance progress: 16.66666666666667
2021/12/14 15:14:12 Rebalance progress: 16.66666666666667
2021/12/14 15:14:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:14:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:14:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:14:17 Waiting for src bucket item count to get to: 200000 curr count: 34749
2021/12/14 15:14:22 Waiting for src bucket item count to get to: 200000 curr count: 50854
2021/12/14 15:14:27 Waiting for src bucket item count to get to: 200000 curr count: 50854
2021/12/14 15:14:32 Waiting for src bucket item count to get to: 200000 curr count: 67261
2021/12/14 15:14:37 Waiting for src bucket item count to get to: 200000 curr count: 67261
2021/12/14 15:14:42 Waiting for src bucket item count to get to: 200000 curr count: 82730
2021/12/14 15:14:47 Waiting for src bucket item count to get to: 200000 curr count: 82730
2021/12/14 15:14:52 Waiting for src bucket item count to get to: 200000 curr count: 99037
2021/12/14 15:14:57 Waiting for src bucket item count to get to: 200000 curr count: 99037
2021/12/14 15:15:02 Waiting for src bucket item count to get to: 200000 curr count: 99037
2021/12/14 15:15:07 Waiting for src bucket item count to get to: 200000 curr count: 115331
2021/12/14 15:15:12 Waiting for src bucket item count to get to: 200000 curr count: 115331
2021/12/14 15:15:17 Waiting for src bucket item count to get to: 200000 curr count: 131347
2021/12/14 15:15:22 Waiting for src bucket item count to get to: 200000 curr count: 131347
2021/12/14 15:15:27 Waiting for src bucket item count to get to: 200000 curr count: 147713
2021/12/14 15:15:32 Waiting for src bucket item count to get to: 200000 curr count: 147713
2021/12/14 15:15:37 Waiting for src bucket item count to get to: 200000 curr count: 164038
2021/12/14 15:15:42 Waiting for src bucket item count to get to: 200000 curr count: 164038
2021/12/14 15:15:47 Waiting for src bucket item count to get to: 200000 curr count: 179283
2021/12/14 15:15:52 Waiting for src bucket item count to get to: 200000 curr count: 179283
2021/12/14 15:15:57 Waiting for src bucket item count to get to: 200000 curr count: 195614
2021/12/14 15:16:02 Waiting for src bucket item count to get to: 200000 curr count: 195614
2021/12/14 15:16:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 15:16:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 15:16:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:16:12 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 15:16:17 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 15:16:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 15:16:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/14 15:16:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/14 15:16:28 No eventing-con process running
2021/12/14 15:16:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/14 15:16:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:16:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:16:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:16:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:16:51 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:16:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:16:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.08s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/14 15:17:01 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 15:17:01 Removing node(s): [] from the cluster
2021/12/14 15:17:06 Rebalance progress: 100
2021/12/14 15:17:11 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 15:17:12 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 15:17:12 No eventing-con process running
2021/12/14 15:17:12 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/14 15:17:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:17:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:17:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:17:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:17:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:17:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:17:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:17:30 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:17:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 15:17:36 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 15:17:41 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 15:17:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 15:17:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/14 15:17:51 App: TestDuplicateWithKVRebOut got deployed
2021/12/14 15:17:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:17:51 Starting bucket ops to source bucket
2021/12/14 15:17:56 Rebalance progress: 16.66666666666667
2021/12/14 15:18:01 Rebalance progress: 16.66666666666667
2021/12/14 15:18:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:18:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:18:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:18:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:18:06 Waiting for src bucket item count to get to: 200000 curr count: 12237
2021/12/14 15:18:11 Waiting for src bucket item count to get to: 200000 curr count: 28561
2021/12/14 15:18:16 Waiting for src bucket item count to get to: 200000 curr count: 28561
2021/12/14 15:18:21 Waiting for src bucket item count to get to: 200000 curr count: 44892
2021/12/14 15:18:26 Waiting for src bucket item count to get to: 200000 curr count: 44892
2021/12/14 15:18:31 Waiting for src bucket item count to get to: 200000 curr count: 60940
2021/12/14 15:18:36 Waiting for src bucket item count to get to: 200000 curr count: 60940
2021/12/14 15:18:41 Waiting for src bucket item count to get to: 200000 curr count: 60940
2021/12/14 15:18:46 Waiting for src bucket item count to get to: 200000 curr count: 76295
2021/12/14 15:18:51 Waiting for src bucket item count to get to: 200000 curr count: 76295
2021/12/14 15:18:56 Waiting for src bucket item count to get to: 200000 curr count: 92426
2021/12/14 15:19:01 Waiting for src bucket item count to get to: 200000 curr count: 92426
2021/12/14 15:19:06 Waiting for src bucket item count to get to: 200000 curr count: 108730
2021/12/14 15:19:11 Waiting for src bucket item count to get to: 200000 curr count: 108730
2021/12/14 15:19:16 Waiting for src bucket item count to get to: 200000 curr count: 125114
2021/12/14 15:19:21 Waiting for src bucket item count to get to: 200000 curr count: 125114
2021/12/14 15:19:26 Waiting for src bucket item count to get to: 200000 curr count: 141567
2021/12/14 15:19:31 Waiting for src bucket item count to get to: 200000 curr count: 141567
2021/12/14 15:19:36 Waiting for src bucket item count to get to: 200000 curr count: 157951
2021/12/14 15:19:41 Waiting for src bucket item count to get to: 200000 curr count: 157951
2021/12/14 15:19:46 Waiting for src bucket item count to get to: 200000 curr count: 173581
2021/12/14 15:19:52 Waiting for src bucket item count to get to: 200000 curr count: 173581
2021/12/14 15:19:57 Waiting for src bucket item count to get to: 200000 curr count: 189631
2021/12/14 15:20:02 Waiting for src bucket item count to get to: 200000 curr count: 189631
2021/12/14 15:20:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 15:20:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 15:20:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:20:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 15:20:17 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 15:20:22 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 15:20:27 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/14 15:20:27 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/14 15:20:27 No eventing-con process running
2021/12/14 15:20:27 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/14 15:20:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:20:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:20:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:20:50 Waiting for dst bucket item count to get to: 0 curr count: 77951
2021/12/14 15:20:55 Waiting for dst bucket item count to get to: 0 curr count: 77951
2021/12/14 15:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (239.19s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/14 15:21:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 15:21:05 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 15:21:05 No eventing-con process running
2021/12/14 15:21:05 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/14 15:21:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:21:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:21:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:21:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:21:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:21:24 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:21:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 15:21:30 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 15:21:35 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 15:21:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 15:21:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/14 15:21:45 App: TestDuplicateWithKVSwapReb got deployed
2021/12/14 15:21:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 15:21:45 Starting bucket ops to source bucket
2021/12/14 15:21:46 Removing node(s): [] from the cluster
2021/12/14 15:21:51 Rebalance progress: 16.66666666666667
2021/12/14 15:21:56 Rebalance progress: 16.66666666666667
2021/12/14 15:22:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:22:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:22:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 15:22:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:22:06 Rebalance progress: 16.66666666666667
2021/12/14 15:22:11 Rebalance progress: 16.66666666666667
2021/12/14 15:22:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:22:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:22:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:22:21 Rebalance progress: 16.66666666666667
2021/12/14 15:22:26 Rebalance progress: 16.66666666666667
2021/12/14 15:22:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:22:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:22:31 Waiting for src bucket item count to get to: 200000 curr count: 68670
2021/12/14 15:22:36 Waiting for src bucket item count to get to: 200000 curr count: 68670
2021/12/14 15:22:41 Waiting for src bucket item count to get to: 200000 curr count: 83561
2021/12/14 15:22:46 Waiting for src bucket item count to get to: 200000 curr count: 83561
2021/12/14 15:22:51 Waiting for src bucket item count to get to: 200000 curr count: 99783
2021/12/14 15:22:56 Waiting for src bucket item count to get to: 200000 curr count: 99783
2021/12/14 15:23:01 Waiting for src bucket item count to get to: 200000 curr count: 116017
2021/12/14 15:23:06 Waiting for src bucket item count to get to: 200000 curr count: 116017
2021/12/14 15:23:11 Waiting for src bucket item count to get to: 200000 curr count: 132193
2021/12/14 15:23:16 Waiting for src bucket item count to get to: 200000 curr count: 132193
2021/12/14 15:23:21 Waiting for src bucket item count to get to: 200000 curr count: 148481
2021/12/14 15:23:26 Waiting for src bucket item count to get to: 200000 curr count: 148481
2021/12/14 15:23:31 Waiting for src bucket item count to get to: 200000 curr count: 164838
2021/12/14 15:23:36 Waiting for src bucket item count to get to: 200000 curr count: 164838
2021/12/14 15:23:41 Waiting for src bucket item count to get to: 200000 curr count: 179926
2021/12/14 15:23:46 Waiting for src bucket item count to get to: 200000 curr count: 179926
2021/12/14 15:23:51 Waiting for src bucket item count to get to: 200000 curr count: 179926
2021/12/14 15:23:56 Waiting for src bucket item count to get to: 200000 curr count: 196027
2021/12/14 15:24:01 Waiting for src bucket item count to get to: 200000 curr count: 196027
2021/12/14 15:24:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 15:24:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 15:24:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:24:12 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 15:24:17 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 15:24:22 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 15:24:27 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/14 15:24:27 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/14 15:24:27 No eventing-con process running
2021/12/14 15:24:27 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/14 15:24:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:24:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:24:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/14 15:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:24:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:24:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:24:50 Waiting for dst bucket item count to get to: 0 curr count: 61925
2021/12/14 15:24:55 Waiting for dst bucket item count to get to: 0 curr count: 61925
2021/12/14 15:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.93s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/14 15:25:05 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:25:05 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:25:05 No eventing-con process running
2021/12/14 15:25:05 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 15:25:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:25:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:25:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:25:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:25:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:25:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:25:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:25:24 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:25:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 15:25:30 Starting bucket ops to source bucket
2021/12/14 15:25:35 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:25:40 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:25:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:25:45 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:25:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:25:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:25:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:25:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:25:50 Rebalance progress: 16.66666666666667
2021/12/14 15:25:55 Rebalance progress: 16.66666666666667
2021/12/14 15:26:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:26:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:26:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:26:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:26:06 Rebalance progress: 16.66666666666667
2021/12/14 15:26:11 Rebalance progress: 16.66666666666667
2021/12/14 15:26:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:26:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/14 15:26:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 15:26:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:26:26 Rebalance progress: 12.5
2021/12/14 15:26:31 Rebalance progress: 12.5
2021/12/14 15:26:36 Rebalance progress: 12.5
2021/12/14 15:26:41 Rebalance progress: 99.2584745762712
2021/12/14 15:26:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:26:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:26:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:26:51 Rebalance progress: 12.5
2021/12/14 15:26:56 Rebalance progress: 12.5
2021/12/14 15:27:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:27:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:27:06 Rebalance progress: 12.5
2021/12/14 15:27:11 Rebalance progress: 12.5
2021/12/14 15:27:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:27:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 15:27:21 Rebalance progress: 12.5
2021/12/14 15:27:26 Rebalance progress: 12.5
2021/12/14 15:27:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:27:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:27:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:27:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:27:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:27:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:27:51 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:27:51 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:27:51 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 15:27:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:27:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:28:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12125
2021/12/14 15:28:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12125
2021/12/14 15:28:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:28:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:28:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:28:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1726
2021/12/14 15:28:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1726
2021/12/14 15:28:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (209.81s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/14 15:28:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 15:28:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 15:28:35 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:28:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/14 15:28:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:28:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:28:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:28:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:28:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:28:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:28:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:28:54 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:28:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/14 15:29:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:29:05 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 15:29:10 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 15:29:15 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/14 15:29:15 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:29:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:29:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:29:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 15:29:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:29:21 Rebalance progress: 12.5
2021/12/14 15:29:26 Rebalance progress: 12.5
2021/12/14 15:29:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:29:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 15:29:36 Rebalance progress: 12.5
2021/12/14 15:29:41 Rebalance progress: 12.5
2021/12/14 15:29:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:29:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:29:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:29:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 15:29:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 15:30:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 15:30:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/14 15:30:06 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/14 15:30:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:30:06 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/14 15:30:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:30:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:30:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4590
2021/12/14 15:30:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:30:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:30:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:30:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.57s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/14 15:30:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 15:30:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 15:30:35 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:30:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/14 15:30:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:30:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:30:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:30:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:30:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:30:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:30:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:30:53 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:30:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/14 15:30:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:31:04 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 15:31:09 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 15:31:14 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/14 15:31:14 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:31:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:31:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:31:20 Rebalance progress: 12.5
2021/12/14 15:31:25 Rebalance progress: 12.5
2021/12/14 15:31:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:31:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:31:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:31:35 Rebalance progress: 12.5
2021/12/14 15:31:40 Rebalance progress: 12.5
2021/12/14 15:31:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:31:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:31:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 15:31:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:31:50 Rebalance progress: 12.5
2021/12/14 15:31:55 Rebalance progress: 12.5
2021/12/14 15:32:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:32:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:32:06 Rebalance progress: 12.5
2021/12/14 15:32:11 Rebalance progress: 12.5
2021/12/14 15:32:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:32:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:32:21 Rebalance progress: 12.5
2021/12/14 15:32:26 Rebalance progress: 12.5
2021/12/14 15:32:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:32:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 15:32:36 Rebalance progress: 12.5
2021/12/14 15:32:41 Rebalance progress: 12.5
2021/12/14 15:32:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:32:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:32:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:32:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 15:32:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 15:33:01 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 15:33:06 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/14 15:33:06 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/14 15:33:06 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:33:06 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/14 15:33:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:33:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:33:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10633
2021/12/14 15:33:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:33:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:33:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:33:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (179.98s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/14 15:33:35 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 15:33:35 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 15:33:35 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:33:35 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/14 15:33:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:33:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:33:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:33:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:33:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:33:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:33:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:33:48 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:33:49 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 15:33:54 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:33:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 15:34:04 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 15:34:09 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/14 15:34:09 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:34:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:34:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:34:15 Rebalance progress: 12.5
2021/12/14 15:34:20 Rebalance progress: 12.5
2021/12/14 15:34:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:34:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:34:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:34:31 Rebalance progress: 12.5
2021/12/14 15:34:36 Rebalance progress: 12.5
2021/12/14 15:34:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:34:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:34:46 Rebalance progress: 12.5
2021/12/14 15:34:51 Rebalance progress: 12.5
2021/12/14 15:34:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:34:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:34:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:35:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 15:35:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 15:35:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 15:35:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/14 15:35:16 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/14 15:35:16 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:35:16 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/14 15:35:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:35:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:35:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6137
2021/12/14 15:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:35:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:35:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:35:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.83s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/14 15:35:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:35:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:35:45 Rebalance progress: 100
2021/12/14 15:35:50 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 15:35:50 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 15:35:50 No eventing-con process running
2021/12/14 15:35:50 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/14 15:35:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:35:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:35:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:36:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:36:03 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:36:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 15:36:10 Starting bucket ops to source bucket
2021/12/14 15:36:15 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 15:36:20 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 15:36:25 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/14 15:36:25 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:36:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:36:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:36:30 Rebalance progress: 12.5
2021/12/14 15:36:35 Rebalance progress: 12.5
2021/12/14 15:36:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:36:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 15:36:40 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 15:36:45 Rebalance progress: 12.5
2021/12/14 15:36:50 Rebalance progress: 12.5
2021/12/14 15:36:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:36:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:36:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:37:00 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 15:37:05 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 15:37:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 15:37:15 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/14 15:37:15 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/14 15:37:15 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:37:15 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/14 15:37:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:37:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:37:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4544
2021/12/14 15:37:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:37:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:37:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:37:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4544
2021/12/14 15:37:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (124.39s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/14 15:37:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:37:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:37:49 Rebalance progress: 100
2021/12/14 15:37:54 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 15:37:54 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 15:37:54 No eventing-con process running
2021/12/14 15:37:54 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/14 15:37:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:38:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:38:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:38:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:38:08 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:38:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/14 15:38:14 Starting bucket ops to source bucket
2021/12/14 15:38:19 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 15:38:24 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 15:38:29 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/14 15:38:29 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:38:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:38:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:38:34 Rebalance progress: 12.5
2021/12/14 15:38:39 Rebalance progress: 12.5
2021/12/14 15:38:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:38:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:38:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 15:38:45 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 15:38:50 Rebalance progress: 12.5
2021/12/14 15:38:55 Rebalance progress: 12.5
2021/12/14 15:39:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:39:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:39:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:39:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:39:00 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/14 15:39:05 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 15:39:10 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 15:39:15 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 15:39:20 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/14 15:39:20 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/14 15:39:20 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:39:20 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/14 15:39:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:39:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:39:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37429
2021/12/14 15:39:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37429
2021/12/14 15:39:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:39:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:39:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:39:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37429
2021/12/14 15:39:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.50s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/14 15:39:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 15:39:58 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 15:39:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:39:58 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/14 15:40:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:40:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:40:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:40:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:40:17 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:40:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:40:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:40:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 15:40:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:40:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 15:40:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/14 15:40:34 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/14 15:40:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 15:40:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 15:40:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 15:40:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/14 15:40:54 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:40:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:40:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:40:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:41:00 Rebalance progress: 12.5
2021/12/14 15:41:05 Rebalance progress: 12.5
2021/12/14 15:41:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:41:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:41:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:41:16 Rebalance progress: 12.5
2021/12/14 15:41:21 Rebalance progress: 12.5
2021/12/14 15:41:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:41:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 15:41:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:41:31 Rebalance progress: 12.5
2021/12/14 15:41:36 Rebalance progress: 12.5
2021/12/14 15:41:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:41:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:41:46 Rebalance progress: 12.5
2021/12/14 15:41:51 Rebalance progress: 12.5
2021/12/14 15:41:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:41:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:41:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:42:01 Rebalance progress: 12.5
2021/12/14 15:42:06 Rebalance progress: 12.5
2021/12/14 15:42:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:42:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:42:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 15:42:16 Rebalance progress: 12.5
2021/12/14 15:42:21 Rebalance progress: 12.5
2021/12/14 15:42:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:42:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:42:31 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 15:42:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 15:42:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 15:42:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/14 15:42:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/14 15:42:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:42:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/14 15:42:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:42:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:42:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:42:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:43:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:43:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12181
2021/12/14 15:43:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:43:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:43:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 15:43:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 15:43:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 15:43:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/14 15:43:27 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/14 15:43:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:43:27 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/14 15:43:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:43:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:43:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:43:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:43:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:43:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.80s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/14 15:43:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 15:43:50 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 15:43:50 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:43:50 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/14 15:43:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:43:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:43:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:43:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:44:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:44:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:44:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:44:09 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:44:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:44:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:44:10 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:44:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 15:44:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:44:21 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 15:44:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 15:44:31 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/14 15:44:31 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/14 15:44:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 15:44:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 15:44:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/14 15:44:46 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:44:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:44:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:44:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:44:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:44:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/14 15:44:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:44:53 Rebalance progress: 12.5
2021/12/14 15:44:58 Rebalance progress: 12.5
2021/12/14 15:45:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:45:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:45:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 15:45:08 Rebalance progress: 12.5
2021/12/14 15:45:13 Rebalance progress: 12.5
2021/12/14 15:45:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:45:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:45:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:45:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 15:45:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 15:45:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 15:45:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/14 15:45:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/14 15:45:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:45:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/14 15:45:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:45:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:45:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2427
2021/12/14 15:45:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2427
2021/12/14 15:45:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:45:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:46:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:46:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6158
2021/12/14 15:46:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:46:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:46:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 15:46:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 15:46:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 15:46:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/14 15:46:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/14 15:46:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:46:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/14 15:46:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:46:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:46:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:46:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:46:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:46:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (182.22s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/14 15:46:52 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 15:46:52 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 15:46:52 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:46:52 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/14 15:46:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:46:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:47:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:47:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:47:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:47:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 15:47:11 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 15:47:11 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:47:11 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/14 15:47:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:47:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:47:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:47:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:47:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:47:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:47:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:47:24 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:47:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:47:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:47:26 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:47:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 15:47:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:47:37 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 15:47:42 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 15:47:47 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/14 15:47:47 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/14 15:47:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 15:47:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 15:48:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 15:48:02 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:48:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:48:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:48:07 Rebalance progress: 12.5
2021/12/14 15:48:12 Rebalance progress: 12.5
2021/12/14 15:48:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:48:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:48:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:48:23 Rebalance progress: 12.5
2021/12/14 15:48:28 Rebalance progress: 12.5
2021/12/14 15:48:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:48:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:48:38 Rebalance progress: 12.5
2021/12/14 15:48:43 Rebalance progress: 12.5
2021/12/14 15:48:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:48:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:48:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 15:48:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 15:49:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 15:49:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 15:49:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 15:49:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:49:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/14 15:49:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:49:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:49:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3145
2021/12/14 15:49:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3145
2021/12/14 15:49:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:49:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:49:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:49:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7593
2021/12/14 15:49:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:49:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:49:43 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 15:49:48 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 15:49:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 15:49:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 15:49:58 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 15:49:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:49:58 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/14 15:50:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:50:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:50:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:50:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:50:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:50:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.68s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 15:50:22 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:50:22 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:50:22 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:50:22 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 15:50:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:50:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:50:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:50:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:50:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:50:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:50:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:50:41 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:50:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 15:50:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:50:52 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:50:57 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:51:02 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:51:02 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:51:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:51:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:51:22 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:51:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:51:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:51:58 Rebalance progress: 12.5
2021/12/14 15:52:03 Rebalance progress: 12.5
2021/12/14 15:52:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:52:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:52:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:52:13 Rebalance progress: 12.5
2021/12/14 15:52:18 Rebalance progress: 12.5
2021/12/14 15:52:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:52:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:52:28 Rebalance progress: 12.5
2021/12/14 15:52:33 Rebalance progress: 12.5
2021/12/14 15:52:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:52:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:52:44 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:52:49 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:52:54 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:52:59 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:52:59 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:52:59 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:52:59 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 15:53:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:53:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:53:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4905
2021/12/14 15:53:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4905
2021/12/14 15:53:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:53:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:53:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:53:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11153
2021/12/14 15:53:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.13s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 15:53:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:53:37 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:53:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:53:37 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 15:53:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:53:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:53:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:53:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:53:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:53:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:53:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:53:56 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:53:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 15:54:02 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:54:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:54:12 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:54:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 15:54:17 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:54:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:54:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:54:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:54:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:54:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:54:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:55:08 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:55:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:55:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:55:40 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:55:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:55:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:55:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:56:11 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:56:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:56:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:56:42 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:56:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:56:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:56:57 Rebalance progress: 12.5
2021/12/14 15:57:02 Rebalance progress: 12.5
2021/12/14 15:57:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:57:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:57:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:57:13 Rebalance progress: 12.5
2021/12/14 15:57:18 Rebalance progress: 12.5
2021/12/14 15:57:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:57:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 15:57:28 Rebalance progress: 12.5
2021/12/14 15:57:33 Rebalance progress: 12.5
2021/12/14 15:57:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:57:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:57:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 15:57:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:57:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:57:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:57:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 15:57:58 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 15:57:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:57:58 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 15:58:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:58:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:58:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:58:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:58:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:58:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21562
2021/12/14 15:58:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (289.35s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/14 15:58:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 15:58:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 15:58:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 15:58:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/14 15:58:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:58:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:58:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 15:58:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:58:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:58:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:58:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 15:58:45 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 15:58:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 15:58:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 15:58:56 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 15:59:01 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 15:59:06 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 15:59:06 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:59:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 15:59:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:59:12 Rebalance progress: 12.5
2021/12/14 15:59:17 Rebalance progress: 12.5
2021/12/14 15:59:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:59:22 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:59:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 15:59:37 Rebalance progress: 12.5
2021/12/14 15:59:42 Rebalance progress: 12.5
2021/12/14 15:59:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 15:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 15:59:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 15:59:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 15:59:53 Rebalance progress: 12.5
2021/12/14 15:59:58 Rebalance progress: 12.5
2021/12/14 16:00:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:00:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 16:00:08 Rebalance progress: 12.5
2021/12/14 16:00:13 Rebalance progress: 12.5
2021/12/14 16:00:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:00:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:00:23 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 16:00:28 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 16:00:33 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 16:00:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 16:00:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 16:00:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 16:00:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/14 16:00:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:00:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:00:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1593
2021/12/14 16:00:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1593
2021/12/14 16:00:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:00:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:01:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8684
2021/12/14 16:01:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (170.36s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/14 16:01:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 16:01:17 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 16:01:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 16:01:17 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/14 16:01:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:01:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:01:35 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:01:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 16:01:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 16:01:47 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 16:01:52 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 16:01:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 16:01:57 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:01:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:01:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/14 16:01:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 16:02:02 Rebalance progress: 12.5
2021/12/14 16:02:07 Rebalance progress: 12.5
2021/12/14 16:02:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:02:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/14 16:02:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:02:18 Rebalance progress: 12.5
2021/12/14 16:02:23 Rebalance progress: 12.5
2021/12/14 16:02:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:02:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 16:02:33 Rebalance progress: 12.5
2021/12/14 16:02:38 Rebalance progress: 12.5
2021/12/14 16:02:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:02:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:02:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:02:48 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 16:02:53 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 16:02:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 16:03:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 16:03:03 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 16:03:03 eventingSetup.go:46: No eventing-con process running
2021/12/14 16:03:04 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/14 16:03:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:03:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:03:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6149
2021/12/14 16:03:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:03:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:03:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:03:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6149
2021/12/14 16:03:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (140.15s)
=== RUN   TestEventingRebInWithOps
2021/12/14 16:03:37 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 16:03:37 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/14 16:03:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 16:03:37 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/14 16:03:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:03:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:03:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:03:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:03:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:03:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:03:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:03:56 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:03:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:04:02 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 16:04:07 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 16:04:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 16:04:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 16:04:17 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/14 16:04:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:04:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 16:04:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 16:04:22 Rebalance progress: 12.5
2021/12/14 16:04:27 Rebalance progress: 12.5
2021/12/14 16:04:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:04:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:04:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:04:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:04:37 Rebalance progress: 12.5
2021/12/14 16:04:42 Rebalance progress: 12.5
2021/12/14 16:04:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:04:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:04:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:04:47 Waiting for src bucket item count to get to: 200000 curr count: 33947
2021/12/14 16:04:52 Waiting for src bucket item count to get to: 200000 curr count: 49603
2021/12/14 16:04:57 Waiting for src bucket item count to get to: 200000 curr count: 49603
2021/12/14 16:05:02 Waiting for src bucket item count to get to: 200000 curr count: 65463
2021/12/14 16:05:07 Waiting for src bucket item count to get to: 200000 curr count: 65463
2021/12/14 16:05:12 Waiting for src bucket item count to get to: 200000 curr count: 80474
2021/12/14 16:05:17 Waiting for src bucket item count to get to: 200000 curr count: 80474
2021/12/14 16:05:22 Waiting for src bucket item count to get to: 200000 curr count: 80474
2021/12/14 16:05:27 Waiting for src bucket item count to get to: 200000 curr count: 96463
2021/12/14 16:05:32 Waiting for src bucket item count to get to: 200000 curr count: 96463
2021/12/14 16:05:37 Waiting for src bucket item count to get to: 200000 curr count: 112612
2021/12/14 16:05:42 Waiting for src bucket item count to get to: 200000 curr count: 112612
2021/12/14 16:05:47 Waiting for src bucket item count to get to: 200000 curr count: 128827
2021/12/14 16:05:52 Waiting for src bucket item count to get to: 200000 curr count: 128827
2021/12/14 16:05:57 Waiting for src bucket item count to get to: 200000 curr count: 144927
2021/12/14 16:06:02 Waiting for src bucket item count to get to: 200000 curr count: 144927
2021/12/14 16:06:07 Waiting for src bucket item count to get to: 200000 curr count: 161194
2021/12/14 16:06:12 Waiting for src bucket item count to get to: 200000 curr count: 161194
2021/12/14 16:06:17 Waiting for src bucket item count to get to: 200000 curr count: 175947
2021/12/14 16:06:22 Waiting for src bucket item count to get to: 200000 curr count: 175947
2021/12/14 16:06:27 Waiting for src bucket item count to get to: 200000 curr count: 191987
2021/12/14 16:06:32 Waiting for src bucket item count to get to: 200000 curr count: 191987
2021/12/14 16:06:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:06:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:06:37 Starting bucket ops to source bucket
2021/12/14 16:10:28 Waiting for dst bucket item count to get to: 0 curr count: 7222
2021/12/14 16:10:33 Waiting for dst bucket item count to get to: 0 curr count: 7222
2021/12/14 16:10:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:10:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:10:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:10:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 16:10:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 16:10:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 16:10:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 16:10:53 App: TestEventingRebInWithOps got un-deployed
2021/12/14 16:10:53 No eventing-con process running
2021/12/14 16:10:54 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/14 16:10:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:11:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:11:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:11:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:11:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (460.13s)
=== RUN   TestEventingRebOutWithOps
2021/12/14 16:11:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:11:13 Removing node(s): [] from the cluster
2021/12/14 16:11:18 Rebalance progress: 100
2021/12/14 16:11:23 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 16:11:23 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 16:11:23 No eventing-con process running
2021/12/14 16:11:23 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/14 16:11:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:11:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:11:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:11:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:11:41 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:11:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:11:47 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 16:11:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 16:11:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 16:12:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 16:12:02 App: TestEventingRebOutWithOps got deployed
2021/12/14 16:12:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:12:02 Starting bucket ops to source bucket
2021/12/14 16:12:07 Rebalance progress: 12.5
2021/12/14 16:12:12 Rebalance progress: 12.5
2021/12/14 16:12:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:12:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:12:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:12:18 Waiting for src bucket item count to get to: 200000 curr count: 9793
2021/12/14 16:12:23 Waiting for src bucket item count to get to: 200000 curr count: 25286
2021/12/14 16:12:28 Waiting for src bucket item count to get to: 200000 curr count: 25286
2021/12/14 16:12:33 Waiting for src bucket item count to get to: 200000 curr count: 41225
2021/12/14 16:12:38 Waiting for src bucket item count to get to: 200000 curr count: 41225
2021/12/14 16:12:43 Waiting for src bucket item count to get to: 200000 curr count: 41225
2021/12/14 16:12:48 Waiting for src bucket item count to get to: 200000 curr count: 57411
2021/12/14 16:12:53 Waiting for src bucket item count to get to: 200000 curr count: 57411
2021/12/14 16:12:58 Waiting for src bucket item count to get to: 200000 curr count: 72637
2021/12/14 16:13:03 Waiting for src bucket item count to get to: 200000 curr count: 72637
2021/12/14 16:13:08 Waiting for src bucket item count to get to: 200000 curr count: 88469
2021/12/14 16:13:13 Waiting for src bucket item count to get to: 200000 curr count: 88469
2021/12/14 16:13:18 Waiting for src bucket item count to get to: 200000 curr count: 104474
2021/12/14 16:13:23 Waiting for src bucket item count to get to: 200000 curr count: 104474
2021/12/14 16:13:28 Waiting for src bucket item count to get to: 200000 curr count: 120516
2021/12/14 16:13:33 Waiting for src bucket item count to get to: 200000 curr count: 120516
2021/12/14 16:13:38 Waiting for src bucket item count to get to: 200000 curr count: 136797
2021/12/14 16:13:43 Waiting for src bucket item count to get to: 200000 curr count: 136797
2021/12/14 16:13:48 Waiting for src bucket item count to get to: 200000 curr count: 152831
2021/12/14 16:13:53 Waiting for src bucket item count to get to: 200000 curr count: 152831
2021/12/14 16:13:58 Waiting for src bucket item count to get to: 200000 curr count: 168005
2021/12/14 16:14:03 Waiting for src bucket item count to get to: 200000 curr count: 168005
2021/12/14 16:14:08 Waiting for src bucket item count to get to: 200000 curr count: 183704
2021/12/14 16:14:13 Waiting for src bucket item count to get to: 200000 curr count: 183704
2021/12/14 16:14:18 Waiting for src bucket item count to get to: 200000 curr count: 199966
2021/12/14 16:14:23 Waiting for src bucket item count to get to: 200000 curr count: 199966
2021/12/14 16:14:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:14:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:14:28 Starting bucket ops to source bucket
2021/12/14 16:18:19 Waiting for dst bucket item count to get to: 0 curr count: 7671
2021/12/14 16:18:24 Waiting for dst bucket item count to get to: 0 curr count: 7671
2021/12/14 16:18:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:18:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:18:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:18:29 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 16:18:34 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 16:18:39 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 16:18:44 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 16:18:44 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 16:18:44 No eventing-con process running
2021/12/14 16:18:44 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/14 16:18:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:18:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:18:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:18:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:18:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:18:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (470.10s)
=== RUN   TestEventingSwapRebWithOps
2021/12/14 16:19:07 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 16:19:07 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 16:19:07 No eventing-con process running
2021/12/14 16:19:07 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/14 16:19:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:19:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:19:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:19:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:19:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:19:26 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:19:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:19:32 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 16:19:37 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 16:19:42 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 16:19:47 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 16:19:47 App: TestEventingSwapRebWithOps got deployed
2021/12/14 16:19:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:19:47 Starting bucket ops to source bucket
2021/12/14 16:19:47 Removing node(s): [] from the cluster
2021/12/14 16:19:52 Rebalance progress: 12.5
2021/12/14 16:19:57 Rebalance progress: 12.5
2021/12/14 16:20:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:20:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 16:20:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:20:08 Rebalance progress: 12.5
2021/12/14 16:20:13 Rebalance progress: 12.5
2021/12/14 16:20:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:20:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 16:20:23 Rebalance progress: 12.5
2021/12/14 16:20:28 Rebalance progress: 12.5
2021/12/14 16:20:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:20:33 Waiting for src bucket item count to get to: 200000 curr count: 64408
2021/12/14 16:20:38 Waiting for src bucket item count to get to: 200000 curr count: 64408
2021/12/14 16:20:44 Waiting for src bucket item count to get to: 200000 curr count: 78643
2021/12/14 16:20:49 Waiting for src bucket item count to get to: 200000 curr count: 78643
2021/12/14 16:20:54 Waiting for src bucket item count to get to: 200000 curr count: 94911
2021/12/14 16:20:59 Waiting for src bucket item count to get to: 200000 curr count: 94911
2021/12/14 16:21:04 Waiting for src bucket item count to get to: 200000 curr count: 111167
2021/12/14 16:21:09 Waiting for src bucket item count to get to: 200000 curr count: 111167
2021/12/14 16:21:14 Waiting for src bucket item count to get to: 200000 curr count: 127156
2021/12/14 16:21:19 Waiting for src bucket item count to get to: 200000 curr count: 127156
2021/12/14 16:21:24 Waiting for src bucket item count to get to: 200000 curr count: 143244
2021/12/14 16:21:29 Waiting for src bucket item count to get to: 200000 curr count: 143244
2021/12/14 16:21:34 Waiting for src bucket item count to get to: 200000 curr count: 159665
2021/12/14 16:21:39 Waiting for src bucket item count to get to: 200000 curr count: 159665
2021/12/14 16:21:44 Waiting for src bucket item count to get to: 200000 curr count: 174356
2021/12/14 16:21:49 Waiting for src bucket item count to get to: 200000 curr count: 174356
2021/12/14 16:21:54 Waiting for src bucket item count to get to: 200000 curr count: 190488
2021/12/14 16:21:59 Waiting for src bucket item count to get to: 200000 curr count: 190488
2021/12/14 16:22:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:22:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:22:04 Starting bucket ops to source bucket
2021/12/14 16:25:54 Waiting for dst bucket item count to get to: 0 curr count: 17999
2021/12/14 16:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:25:54 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:25:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:25:59 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 16:26:04 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 16:26:09 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 16:26:14 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 16:26:14 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 16:26:14 No eventing-con process running
2021/12/14 16:26:14 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/14 16:26:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:26:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:26:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:26:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (450.74s)
=== RUN   TestKVRebInWithOps
2021/12/14 16:26:38 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 16:26:38 App: TestKVRebInWithOps got un-deployed
2021/12/14 16:26:38 No eventing-con process running
2021/12/14 16:26:38 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/14 16:26:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:26:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:26:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:26:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:26:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:26:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:26:57 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:26:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:27:03 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 16:27:08 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 16:27:13 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 16:27:18 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 16:27:18 App: TestKVRebInWithOps got deployed
2021/12/14 16:27:18 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 16:27:18 Starting bucket ops to source bucket
2021/12/14 16:27:18 Removing node(s): [] from the cluster
2021/12/14 16:27:23 Rebalance progress: 12.5
2021/12/14 16:27:28 Rebalance progress: 12.5
2021/12/14 16:27:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:27:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:27:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:27:38 Rebalance progress: 12.5
2021/12/14 16:27:43 Rebalance progress: 12.5
2021/12/14 16:27:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:27:48 Waiting for src bucket item count to get to: 200000 curr count: 32291
2021/12/14 16:27:53 Waiting for src bucket item count to get to: 200000 curr count: 48341
2021/12/14 16:27:58 Waiting for src bucket item count to get to: 200000 curr count: 48341
2021/12/14 16:28:03 Waiting for src bucket item count to get to: 200000 curr count: 64479
2021/12/14 16:28:08 Waiting for src bucket item count to get to: 200000 curr count: 64479
2021/12/14 16:28:13 Waiting for src bucket item count to get to: 200000 curr count: 79458
2021/12/14 16:28:18 Waiting for src bucket item count to get to: 200000 curr count: 79458
2021/12/14 16:28:23 Waiting for src bucket item count to get to: 200000 curr count: 95595
2021/12/14 16:28:28 Waiting for src bucket item count to get to: 200000 curr count: 95595
2021/12/14 16:28:33 Waiting for src bucket item count to get to: 200000 curr count: 95595
2021/12/14 16:28:38 Waiting for src bucket item count to get to: 200000 curr count: 111710
2021/12/14 16:28:43 Waiting for src bucket item count to get to: 200000 curr count: 111710
2021/12/14 16:28:48 Waiting for src bucket item count to get to: 200000 curr count: 127844
2021/12/14 16:28:53 Waiting for src bucket item count to get to: 200000 curr count: 127844
2021/12/14 16:28:58 Waiting for src bucket item count to get to: 200000 curr count: 143840
2021/12/14 16:29:03 Waiting for src bucket item count to get to: 200000 curr count: 143840
2021/12/14 16:29:08 Waiting for src bucket item count to get to: 200000 curr count: 159759
2021/12/14 16:29:13 Waiting for src bucket item count to get to: 200000 curr count: 159759
2021/12/14 16:29:18 Waiting for src bucket item count to get to: 200000 curr count: 174771
2021/12/14 16:29:23 Waiting for src bucket item count to get to: 200000 curr count: 174771
2021/12/14 16:29:28 Waiting for src bucket item count to get to: 200000 curr count: 190517
2021/12/14 16:29:33 Waiting for src bucket item count to get to: 200000 curr count: 190517
2021/12/14 16:29:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:29:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:29:38 Starting bucket ops to source bucket
2021/12/14 16:33:29 Waiting for dst bucket item count to get to: 0 curr count: 7841
2021/12/14 16:33:34 Waiting for dst bucket item count to get to: 0 curr count: 7841
2021/12/14 16:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:33:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:33:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:33:39 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 16:33:44 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 16:33:49 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 16:33:54 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 16:33:54 App: TestKVRebInWithOps got un-deployed
2021/12/14 16:33:54 No eventing-con process running
2021/12/14 16:33:54 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/14 16:33:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:34:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:34:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:34:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:34:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.39s)
=== RUN   TestKVRebOutWithOps
2021/12/14 16:34:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 16:34:13 Removing node(s): [] from the cluster
2021/12/14 16:34:18 Rebalance progress: 100
2021/12/14 16:34:23 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 16:34:23 App: TestKVRebOutWithOps got un-deployed
2021/12/14 16:34:23 No eventing-con process running
2021/12/14 16:34:23 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/14 16:34:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:34:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:34:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:34:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:34:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:34:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:34:42 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:34:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:34:48 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 16:34:53 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 16:34:58 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 16:35:03 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 16:35:03 App: TestKVRebOutWithOps got deployed
2021/12/14 16:35:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:35:03 Starting bucket ops to source bucket
2021/12/14 16:35:08 Rebalance progress: 12.5
2021/12/14 16:35:13 Rebalance progress: 12.5
2021/12/14 16:35:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:35:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:35:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:35:18 Waiting for src bucket item count to get to: 200000 curr count: 9291
2021/12/14 16:35:23 Waiting for src bucket item count to get to: 200000 curr count: 25034
2021/12/14 16:35:28 Waiting for src bucket item count to get to: 200000 curr count: 25034
2021/12/14 16:35:33 Waiting for src bucket item count to get to: 200000 curr count: 41071
2021/12/14 16:35:38 Waiting for src bucket item count to get to: 200000 curr count: 41071
2021/12/14 16:35:43 Waiting for src bucket item count to get to: 200000 curr count: 57132
2021/12/14 16:35:48 Waiting for src bucket item count to get to: 200000 curr count: 57132
2021/12/14 16:35:53 Waiting for src bucket item count to get to: 200000 curr count: 57132
2021/12/14 16:35:58 Waiting for src bucket item count to get to: 200000 curr count: 72440
2021/12/14 16:36:03 Waiting for src bucket item count to get to: 200000 curr count: 72440
2021/12/14 16:36:08 Waiting for src bucket item count to get to: 200000 curr count: 87895
2021/12/14 16:36:13 Waiting for src bucket item count to get to: 200000 curr count: 87895
2021/12/14 16:36:18 Waiting for src bucket item count to get to: 200000 curr count: 103964
2021/12/14 16:36:23 Waiting for src bucket item count to get to: 200000 curr count: 103964
2021/12/14 16:36:28 Waiting for src bucket item count to get to: 200000 curr count: 119904
2021/12/14 16:36:33 Waiting for src bucket item count to get to: 200000 curr count: 119904
2021/12/14 16:36:38 Waiting for src bucket item count to get to: 200000 curr count: 135814
2021/12/14 16:36:43 Waiting for src bucket item count to get to: 200000 curr count: 135814
2021/12/14 16:36:48 Waiting for src bucket item count to get to: 200000 curr count: 151969
2021/12/14 16:36:53 Waiting for src bucket item count to get to: 200000 curr count: 151969
2021/12/14 16:36:58 Waiting for src bucket item count to get to: 200000 curr count: 167426
2021/12/14 16:37:03 Waiting for src bucket item count to get to: 200000 curr count: 167426
2021/12/14 16:37:08 Waiting for src bucket item count to get to: 200000 curr count: 182483
2021/12/14 16:37:13 Waiting for src bucket item count to get to: 200000 curr count: 182483
2021/12/14 16:37:18 Waiting for src bucket item count to get to: 200000 curr count: 198627
2021/12/14 16:37:23 Waiting for src bucket item count to get to: 200000 curr count: 198627
2021/12/14 16:37:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:37:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:37:28 Starting bucket ops to source bucket
2021/12/14 16:41:19 Waiting for dst bucket item count to get to: 0 curr count: 7985
2021/12/14 16:41:24 Waiting for dst bucket item count to get to: 0 curr count: 7985
2021/12/14 16:41:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:41:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:41:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:41:29 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 16:41:34 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 16:41:39 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 16:41:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 16:41:44 App: TestKVRebOutWithOps got un-deployed
2021/12/14 16:41:44 No eventing-con process running
2021/12/14 16:41:44 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/14 16:41:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:41:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:41:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:41:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:41:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (470.11s)
=== RUN   TestKVSwapRebWithOps
2021/12/14 16:42:08 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 16:42:08 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 16:42:08 No eventing-con process running
2021/12/14 16:42:08 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/14 16:42:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:42:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:42:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:42:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:42:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:42:26 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:42:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 16:42:32 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 16:42:37 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 16:42:42 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 16:42:47 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 16:42:47 App: TestKVSwapRebWithOps got deployed
2021/12/14 16:42:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 16:42:47 Starting bucket ops to source bucket
2021/12/14 16:42:48 Removing node(s): [] from the cluster
2021/12/14 16:42:53 Rebalance progress: 12.5
2021/12/14 16:42:58 Rebalance progress: 12.5
2021/12/14 16:43:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:43:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 16:43:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:43:08 Rebalance progress: 12.5
2021/12/14 16:43:13 Rebalance progress: 12.5
2021/12/14 16:43:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:43:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 16:43:23 Rebalance progress: 12.5
2021/12/14 16:43:28 Rebalance progress: 12.5
2021/12/14 16:43:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:43:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:43:33 Waiting for src bucket item count to get to: 200000 curr count: 63755
2021/12/14 16:43:38 Waiting for src bucket item count to get to: 200000 curr count: 63755
2021/12/14 16:43:43 Waiting for src bucket item count to get to: 200000 curr count: 78178
2021/12/14 16:43:48 Waiting for src bucket item count to get to: 200000 curr count: 78178
2021/12/14 16:43:53 Waiting for src bucket item count to get to: 200000 curr count: 94361
2021/12/14 16:43:58 Waiting for src bucket item count to get to: 200000 curr count: 94361
2021/12/14 16:44:03 Waiting for src bucket item count to get to: 200000 curr count: 110447
2021/12/14 16:44:08 Waiting for src bucket item count to get to: 200000 curr count: 110447
2021/12/14 16:44:13 Waiting for src bucket item count to get to: 200000 curr count: 126485
2021/12/14 16:44:18 Waiting for src bucket item count to get to: 200000 curr count: 126485
2021/12/14 16:44:23 Waiting for src bucket item count to get to: 200000 curr count: 142895
2021/12/14 16:44:28 Waiting for src bucket item count to get to: 200000 curr count: 142895
2021/12/14 16:44:33 Waiting for src bucket item count to get to: 200000 curr count: 159021
2021/12/14 16:44:38 Waiting for src bucket item count to get to: 200000 curr count: 159021
2021/12/14 16:44:43 Waiting for src bucket item count to get to: 200000 curr count: 159021
2021/12/14 16:44:48 Waiting for src bucket item count to get to: 200000 curr count: 173837
2021/12/14 16:44:53 Waiting for src bucket item count to get to: 200000 curr count: 173837
2021/12/14 16:44:58 Waiting for src bucket item count to get to: 200000 curr count: 189720
2021/12/14 16:45:03 Waiting for src bucket item count to get to: 200000 curr count: 189720
2021/12/14 16:45:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 16:45:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 16:45:08 Starting bucket ops to source bucket
2021/12/14 16:49:00 Waiting for dst bucket item count to get to: 0 curr count: 9783
2021/12/14 16:49:05 Waiting for dst bucket item count to get to: 0 curr count: 9783
2021/12/14 16:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:49:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:49:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:49:10 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 16:49:15 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 16:49:20 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 16:49:25 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 16:49:25 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 16:49:25 No eventing-con process running
2021/12/14 16:49:25 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/14 16:49:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:49:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:49:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:49:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:49:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (461.08s)
=== RUN   TestEventingRebInWithTimers
2021/12/14 16:49:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 16:49:49 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 16:49:49 No eventing-con process running
2021/12/14 16:49:49 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/14 16:49:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:49:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:49:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:50:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:50:07 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:50:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 16:50:13 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 16:50:18 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 16:50:23 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 16:50:28 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 16:50:28 App: TestEventingRebInWithTimers got deployed
2021/12/14 16:50:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:50:28 Starting bucket ops to source bucket
2021/12/14 16:50:29 Removing node(s): [] from the cluster
2021/12/14 16:50:34 Rebalance progress: 12.5
2021/12/14 16:50:39 Rebalance progress: 12.5
2021/12/14 16:50:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:50:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:50:49 Rebalance progress: 12.5
2021/12/14 16:50:54 Rebalance progress: 12.5
2021/12/14 16:50:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:50:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:50:59 Waiting for src bucket item count to get to: 50000 curr count: 27414
2021/12/14 16:51:04 Waiting for src bucket item count to get to: 50000 curr count: 36773
2021/12/14 16:51:09 Waiting for src bucket item count to get to: 50000 curr count: 36773
2021/12/14 16:51:14 Waiting for src bucket item count to get to: 50000 curr count: 49273
2021/12/14 16:51:19 Waiting for src bucket item count to get to: 50000 curr count: 49273
2021/12/14 16:51:29 Waiting for dst bucket item count to get to: 50000 curr count: 41482
2021/12/14 16:51:34 Waiting for dst bucket item count to get to: 50000 curr count: 41482
2021/12/14 16:51:39 Waiting for dst bucket item count to get to: 50000 curr count: 49786
2021/12/14 16:51:44 Waiting for dst bucket item count to get to: 50000 curr count: 49786
2021/12/14 16:51:49 Waiting for dst bucket item count to get to: 50000 curr count: 49786
2021/12/14 16:51:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 16:51:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 16:51:49 Starting bucket ops to source bucket
2021/12/14 16:53:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:53:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:53:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:53:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:53:20 Waiting for dst bucket item count to get to: 0 curr count: 41063
2021/12/14 16:53:25 Waiting for dst bucket item count to get to: 0 curr count: 41063
2021/12/14 16:53:30 Waiting for dst bucket item count to get to: 0 curr count: 23106
2021/12/14 16:53:35 Waiting for dst bucket item count to get to: 0 curr count: 23106
2021/12/14 16:53:40 Waiting for dst bucket item count to get to: 0 curr count: 11760
2021/12/14 16:53:45 Waiting for dst bucket item count to get to: 0 curr count: 11760
2021/12/14 16:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:53:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:53:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:53:50 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 16:53:55 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 16:54:00 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 16:54:05 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 16:54:05 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 16:54:05 No eventing-con process running
2021/12/14 16:54:05 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/14 16:54:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:54:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:54:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (279.85s)
=== RUN   TestEventingRebOutWithTimers
2021/12/14 16:54:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:54:24 Removing node(s): [] from the cluster
2021/12/14 16:54:29 Rebalance progress: 100
2021/12/14 16:54:34 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 16:54:34 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 16:54:34 No eventing-con process running
2021/12/14 16:54:34 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/14 16:54:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:54:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:54:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:54:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:54:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:54:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:54:52 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:54:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 16:54:58 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 16:55:04 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 16:55:09 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 16:55:14 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 16:55:14 App: TestEventingRebOutWithTimers got deployed
2021/12/14 16:55:14 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 16:55:14 Starting bucket ops to source bucket
2021/12/14 16:55:19 Rebalance progress: 12.5
2021/12/14 16:55:24 Rebalance progress: 12.5
2021/12/14 16:55:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:55:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:55:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 16:55:29 Waiting for src bucket item count to get to: 50000 curr count: 7230
2021/12/14 16:55:34 Waiting for src bucket item count to get to: 50000 curr count: 20636
2021/12/14 16:55:39 Waiting for src bucket item count to get to: 50000 curr count: 20636
2021/12/14 16:55:44 Waiting for src bucket item count to get to: 50000 curr count: 32507
2021/12/14 16:55:49 Waiting for src bucket item count to get to: 50000 curr count: 32507
2021/12/14 16:55:54 Waiting for src bucket item count to get to: 50000 curr count: 32507
2021/12/14 16:55:59 Waiting for src bucket item count to get to: 50000 curr count: 43411
2021/12/14 16:56:04 Waiting for src bucket item count to get to: 50000 curr count: 43411
2021/12/14 16:56:14 Waiting for dst bucket item count to get to: 50000 curr count: 37455
2021/12/14 16:56:19 Waiting for dst bucket item count to get to: 50000 curr count: 37455
2021/12/14 16:56:24 Waiting for dst bucket item count to get to: 50000 curr count: 47622
2021/12/14 16:56:29 Waiting for dst bucket item count to get to: 50000 curr count: 47622
2021/12/14 16:56:29 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 16:56:29 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 16:56:29 Starting bucket ops to source bucket
2021/12/14 16:57:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:57:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:57:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:57:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 16:57:59 Waiting for dst bucket item count to get to: 0 curr count: 35583
2021/12/14 16:58:04 Waiting for dst bucket item count to get to: 0 curr count: 35583
2021/12/14 16:58:09 Waiting for dst bucket item count to get to: 0 curr count: 24617
2021/12/14 16:58:14 Waiting for dst bucket item count to get to: 0 curr count: 24617
2021/12/14 16:58:19 Waiting for dst bucket item count to get to: 0 curr count: 4633
2021/12/14 16:58:24 Waiting for dst bucket item count to get to: 0 curr count: 4633
2021/12/14 16:58:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:58:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 16:58:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 16:58:29 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 16:58:34 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 16:58:39 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 16:58:44 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 16:58:44 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 16:58:44 No eventing-con process running
2021/12/14 16:58:44 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/14 16:58:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:58:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:58:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:58:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:58:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:58:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.14s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/14 16:59:08 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 16:59:08 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 16:59:08 No eventing-con process running
2021/12/14 16:59:08 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/14 16:59:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:59:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:59:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 16:59:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:59:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 16:59:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 16:59:26 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 16:59:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 16:59:32 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 16:59:37 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 16:59:42 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 16:59:47 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 16:59:47 App: TestEventingSwapRebWithTimers got deployed
2021/12/14 16:59:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 16:59:47 Starting bucket ops to source bucket
2021/12/14 16:59:48 Removing node(s): [] from the cluster
2021/12/14 16:59:53 Rebalance progress: 12.5
2021/12/14 16:59:58 Rebalance progress: 12.5
2021/12/14 17:00:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:00:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 17:00:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:00:09 Rebalance progress: 12.5
2021/12/14 17:00:14 Rebalance progress: 12.5
2021/12/14 17:00:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:00:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:00:24 Rebalance progress: 12.5
2021/12/14 17:00:29 Rebalance progress: 12.5
2021/12/14 17:00:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:00:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:00:34 Waiting for src bucket item count to get to: 50000 curr count: 49659
2021/12/14 17:00:39 Waiting for src bucket item count to get to: 50000 curr count: 49659
2021/12/14 17:00:49 Waiting for dst bucket item count to get to: 50000 curr count: 43038
2021/12/14 17:00:54 Waiting for dst bucket item count to get to: 50000 curr count: 43038
2021/12/14 17:00:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:00:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:00:54 Starting bucket ops to source bucket
2021/12/14 17:02:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:02:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:02:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:02:20 Waiting for dst bucket item count to get to: 0 curr count: 43053
2021/12/14 17:02:25 Waiting for dst bucket item count to get to: 0 curr count: 43053
2021/12/14 17:02:30 Waiting for dst bucket item count to get to: 0 curr count: 33758
2021/12/14 17:02:35 Waiting for dst bucket item count to get to: 0 curr count: 33758
2021/12/14 17:02:40 Waiting for dst bucket item count to get to: 0 curr count: 16236
2021/12/14 17:02:45 Waiting for dst bucket item count to get to: 0 curr count: 16236
2021/12/14 17:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:02:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 17:02:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:02:50 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 17:02:55 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 17:03:00 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 17:03:05 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 17:03:05 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 17:03:05 No eventing-con process running
2021/12/14 17:03:05 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/14 17:03:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:03:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:03:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:03:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:03:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (261.40s)
=== RUN   TestKVRebInWithTimers
2021/12/14 17:03:29 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 17:03:29 App: TestKVRebInWithTimers got un-deployed
2021/12/14 17:03:29 No eventing-con process running
2021/12/14 17:03:29 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/14 17:03:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:03:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:03:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:03:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:03:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:03:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:03:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:03:48 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:03:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 17:03:54 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 17:03:59 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 17:04:04 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 17:04:09 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 17:04:09 App: TestKVRebInWithTimers got deployed
2021/12/14 17:04:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:04:09 Starting bucket ops to source bucket
2021/12/14 17:04:09 Removing node(s): [] from the cluster
2021/12/14 17:04:14 Rebalance progress: 12.5
2021/12/14 17:04:19 Rebalance progress: 12.5
2021/12/14 17:04:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:04:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:04:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:04:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:04:29 Rebalance progress: 12.5
2021/12/14 17:04:34 Rebalance progress: 12.5
2021/12/14 17:04:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:04:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:04:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:04:39 Waiting for src bucket item count to get to: 50000 curr count: 24785
2021/12/14 17:04:44 Waiting for src bucket item count to get to: 50000 curr count: 34991
2021/12/14 17:04:49 Waiting for src bucket item count to get to: 50000 curr count: 34991
2021/12/14 17:04:54 Waiting for src bucket item count to get to: 50000 curr count: 47363
2021/12/14 17:04:59 Waiting for src bucket item count to get to: 50000 curr count: 47363
2021/12/14 17:05:09 Waiting for dst bucket item count to get to: 50000 curr count: 48285
2021/12/14 17:05:14 Waiting for dst bucket item count to get to: 50000 curr count: 48285
2021/12/14 17:05:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:05:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:05:14 Starting bucket ops to source bucket
2021/12/14 17:06:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:06:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:06:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:06:40 Waiting for dst bucket item count to get to: 0 curr count: 39804
2021/12/14 17:06:45 Waiting for dst bucket item count to get to: 0 curr count: 39804
2021/12/14 17:06:50 Waiting for dst bucket item count to get to: 0 curr count: 24708
2021/12/14 17:06:55 Waiting for dst bucket item count to get to: 0 curr count: 24708
2021/12/14 17:07:00 Waiting for dst bucket item count to get to: 0 curr count: 11290
2021/12/14 17:07:05 Waiting for dst bucket item count to get to: 0 curr count: 11290
2021/12/14 17:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:07:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 17:07:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:07:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 17:07:15 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 17:07:20 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 17:07:25 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 17:07:25 App: TestKVRebInWithTimers got un-deployed
2021/12/14 17:07:25 No eventing-con process running
2021/12/14 17:07:25 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/14 17:07:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:07:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:07:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:07:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:07:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:07:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.85s)
=== RUN   TestKVRebOutWithTimers
2021/12/14 17:07:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:07:44 Removing node(s): [] from the cluster
2021/12/14 17:07:49 Rebalance progress: 100
2021/12/14 17:07:54 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 17:07:54 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 17:07:54 No eventing-con process running
2021/12/14 17:07:54 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/14 17:07:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:08:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:08:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:08:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:08:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:08:13 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:08:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 17:08:19 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 17:08:24 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 17:08:29 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 17:08:34 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 17:08:34 App: TestKVRebOutWithTimers got deployed
2021/12/14 17:08:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:08:34 Starting bucket ops to source bucket
2021/12/14 17:08:39 Rebalance progress: 12.5
2021/12/14 17:08:44 Rebalance progress: 12.5
2021/12/14 17:08:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:08:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:08:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:08:49 Waiting for src bucket item count to get to: 50000 curr count: 6768
2021/12/14 17:08:54 Waiting for src bucket item count to get to: 50000 curr count: 20968
2021/12/14 17:08:59 Waiting for src bucket item count to get to: 50000 curr count: 20968
2021/12/14 17:09:04 Waiting for src bucket item count to get to: 50000 curr count: 32779
2021/12/14 17:09:09 Waiting for src bucket item count to get to: 50000 curr count: 32779
2021/12/14 17:09:14 Waiting for src bucket item count to get to: 50000 curr count: 43925
2021/12/14 17:09:19 Waiting for src bucket item count to get to: 50000 curr count: 43925
2021/12/14 17:09:24 Waiting for src bucket item count to get to: 50000 curr count: 43925
2021/12/14 17:09:34 Waiting for dst bucket item count to get to: 50000 curr count: 35609
2021/12/14 17:09:39 Waiting for dst bucket item count to get to: 50000 curr count: 35609
2021/12/14 17:09:44 Waiting for dst bucket item count to get to: 50000 curr count: 43706
2021/12/14 17:09:49 Waiting for dst bucket item count to get to: 50000 curr count: 43706
2021/12/14 17:09:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:09:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:09:49 Starting bucket ops to source bucket
2021/12/14 17:11:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:11:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:11:10 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:11:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:11:20 Waiting for dst bucket item count to get to: 0 curr count: 38228
2021/12/14 17:11:25 Waiting for dst bucket item count to get to: 0 curr count: 38228
2021/12/14 17:11:30 Waiting for dst bucket item count to get to: 0 curr count: 26765
2021/12/14 17:11:35 Waiting for dst bucket item count to get to: 0 curr count: 26765
2021/12/14 17:11:40 Waiting for dst bucket item count to get to: 0 curr count: 11106
2021/12/14 17:11:45 Waiting for dst bucket item count to get to: 0 curr count: 11106
2021/12/14 17:11:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:11:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 17:11:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:11:50 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 17:11:55 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 17:12:00 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 17:12:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 17:12:05 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 17:12:05 No eventing-con process running
2021/12/14 17:12:05 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/14 17:12:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:12:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:12:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:12:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:12:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:12:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.95s)
=== RUN   TestKVSwapRebWithTimers
2021/12/14 17:12:29 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:12:29 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 17:12:29 No eventing-con process running
2021/12/14 17:12:29 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/14 17:12:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:12:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:12:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:12:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:12:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:12:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:12:47 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:12:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 17:12:53 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 17:12:58 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 17:13:03 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 17:13:08 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 17:13:09 App: TestKVSwapRebWithTimers got deployed
2021/12/14 17:13:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:13:09 Starting bucket ops to source bucket
2021/12/14 17:13:10 Removing node(s): [] from the cluster
2021/12/14 17:13:15 Rebalance progress: 12.5
2021/12/14 17:13:20 Rebalance progress: 12.5
2021/12/14 17:13:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:13:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 17:13:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:13:30 Rebalance progress: 12.5
2021/12/14 17:13:35 Rebalance progress: 12.5
2021/12/14 17:13:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:13:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:13:46 Rebalance progress: 12.5
2021/12/14 17:13:51 Rebalance progress: 12.5
2021/12/14 17:13:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:13:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:13:56 Waiting for src bucket item count to get to: 50000 curr count: 46113
2021/12/14 17:14:01 Waiting for src bucket item count to get to: 50000 curr count: 46113
2021/12/14 17:14:11 Waiting for dst bucket item count to get to: 50000 curr count: 46391
2021/12/14 17:14:16 Waiting for dst bucket item count to get to: 50000 curr count: 46391
2021/12/14 17:14:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:14:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:14:16 Starting bucket ops to source bucket
2021/12/14 17:15:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:15:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:15:36 Waiting for dst bucket item count to get to: 0 curr count: 43431
2021/12/14 17:15:41 Waiting for dst bucket item count to get to: 0 curr count: 43431
2021/12/14 17:15:46 Waiting for dst bucket item count to get to: 0 curr count: 28493
2021/12/14 17:15:51 Waiting for dst bucket item count to get to: 0 curr count: 28493
2021/12/14 17:15:56 Waiting for dst bucket item count to get to: 0 curr count: 14560
2021/12/14 17:16:02 Waiting for dst bucket item count to get to: 0 curr count: 14560
2021/12/14 17:16:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:16:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 17:16:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:16:07 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:12 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:17 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:22 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:27 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:32 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 17:16:32 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 17:16:32 No eventing-con process running
2021/12/14 17:16:32 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/14 17:16:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:16:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:16:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:16:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (266.61s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/14 17:16:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:16:55 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 17:16:55 No eventing-con process running
2021/12/14 17:16:55 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/14 17:16:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:17:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:17:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:17:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:17:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:17:14 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:17:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 17:17:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 17:17:25 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 17:17:30 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 17:17:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 17:17:35 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/14 17:17:35 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:17:35 Starting bucket ops to source bucket
2021/12/14 17:17:36 Removing node(s): [] from the cluster
2021/12/14 17:17:41 Rebalance progress: 12.5
2021/12/14 17:17:46 Rebalance progress: 12.5
2021/12/14 17:17:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:17:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:17:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:17:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:17:56 Rebalance progress: 12.5
2021/12/14 17:18:01 Rebalance progress: 12.5
2021/12/14 17:18:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:18:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:18:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:18:06 Waiting for src bucket item count to get to: 50000 curr count: 29218
2021/12/14 17:18:11 Waiting for src bucket item count to get to: 50000 curr count: 40069
2021/12/14 17:18:16 Waiting for src bucket item count to get to: 50000 curr count: 40069
2021/12/14 17:18:21 Waiting for src bucket item count to get to: 50000 curr count: 49832
2021/12/14 17:18:26 Waiting for src bucket item count to get to: 50000 curr count: 49832
2021/12/14 17:18:36 Waiting for dst bucket item count to get to: 50000 curr count: 37482
2021/12/14 17:18:41 Waiting for dst bucket item count to get to: 50000 curr count: 37482
2021/12/14 17:18:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:18:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:18:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:18:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:18:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:18:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:19:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:19:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:19:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 17:19:11 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 17:19:11 No eventing-con process running
2021/12/14 17:19:11 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/14 17:19:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:19:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:19:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:19:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:19:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:19:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:19:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:19:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (169.43s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/14 17:19:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:19:40 Removing node(s): [] from the cluster
2021/12/14 17:19:45 Rebalance progress: 100
2021/12/14 17:19:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:19:50 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 17:19:50 No eventing-con process running
2021/12/14 17:19:50 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/14 17:19:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:19:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:19:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:20:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:20:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:20:09 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:20:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 17:20:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 17:20:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 17:20:25 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 17:20:30 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 17:20:30 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/14 17:20:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:20:30 Starting bucket ops to source bucket
2021/12/14 17:20:35 Rebalance progress: 12.5
2021/12/14 17:20:40 Rebalance progress: 12.5
2021/12/14 17:20:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:20:45 Waiting for src bucket item count to get to: 50000 curr count: 11970
2021/12/14 17:20:50 Waiting for src bucket item count to get to: 50000 curr count: 25795
2021/12/14 17:20:55 Waiting for src bucket item count to get to: 50000 curr count: 25795
2021/12/14 17:21:00 Waiting for src bucket item count to get to: 50000 curr count: 36293
2021/12/14 17:21:05 Waiting for src bucket item count to get to: 50000 curr count: 36293
2021/12/14 17:21:10 Waiting for src bucket item count to get to: 50000 curr count: 36293
2021/12/14 17:21:15 Waiting for src bucket item count to get to: 50000 curr count: 46311
2021/12/14 17:21:20 Waiting for src bucket item count to get to: 50000 curr count: 46311
2021/12/14 17:21:30 Waiting for dst bucket item count to get to: 50000 curr count: 37617
2021/12/14 17:21:35 Waiting for dst bucket item count to get to: 50000 curr count: 37617
2021/12/14 17:21:40 Waiting for dst bucket item count to get to: 50000 curr count: 46569
2021/12/14 17:21:45 Waiting for dst bucket item count to get to: 50000 curr count: 46569
2021/12/14 17:21:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:21:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:21:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:21:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:21:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:22:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:22:05 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:22:10 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:22:15 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 17:22:15 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 17:22:15 No eventing-con process running
2021/12/14 17:22:15 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/14 17:22:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:22:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:22:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:22:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:22:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:22:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (184.17s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/14 17:22:49 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:22:49 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 17:22:49 No eventing-con process running
2021/12/14 17:22:49 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/14 17:22:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:22:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:22:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:22:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:23:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:23:08 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:23:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 17:23:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 17:23:19 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 17:23:24 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 17:23:29 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 17:23:29 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/14 17:23:29 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:23:29 Starting bucket ops to source bucket
2021/12/14 17:23:29 Removing node(s): [] from the cluster
2021/12/14 17:23:34 Rebalance progress: 12.5
2021/12/14 17:23:39 Rebalance progress: 12.5
2021/12/14 17:23:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:23:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 17:23:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:23:49 Rebalance progress: 12.5
2021/12/14 17:23:54 Rebalance progress: 12.5
2021/12/14 17:23:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:23:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:24:05 Rebalance progress: 12.5
2021/12/14 17:24:10 Rebalance progress: 12.5
2021/12/14 17:24:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:24:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:24:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:24:15 Waiting for src bucket item count to get to: 50000 curr count: 47127
2021/12/14 17:24:20 Waiting for src bucket item count to get to: 50000 curr count: 47127
2021/12/14 17:24:30 Waiting for dst bucket item count to get to: 50000 curr count: 35391
2021/12/14 17:24:35 Waiting for dst bucket item count to get to: 50000 curr count: 35391
2021/12/14 17:24:40 Waiting for dst bucket item count to get to: 50000 curr count: 41160
2021/12/14 17:24:45 Waiting for dst bucket item count to get to: 50000 curr count: 41160
2021/12/14 17:24:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:24:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:24:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:24:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:24:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:25:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:25:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:25:10 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:25:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 17:25:15 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 17:25:15 No eventing-con process running
2021/12/14 17:25:15 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/14 17:25:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:25:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:25:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:25:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:25:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:25:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.56s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/14 17:25:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:25:49 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 17:25:49 No eventing-con process running
2021/12/14 17:25:49 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/14 17:25:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:25:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:25:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:26:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:26:07 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:26:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 17:26:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 17:26:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 17:26:23 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 17:26:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 17:26:28 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/14 17:26:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:26:28 Starting bucket ops to source bucket
2021/12/14 17:26:29 Removing node(s): [] from the cluster
2021/12/14 17:26:34 Rebalance progress: 12.5
2021/12/14 17:26:39 Rebalance progress: 12.5
2021/12/14 17:26:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:26:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:26:49 Rebalance progress: 12.5
2021/12/14 17:26:54 Rebalance progress: 12.5
2021/12/14 17:26:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:26:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:26:59 Waiting for src bucket item count to get to: 50000 curr count: 26226
2021/12/14 17:27:04 Waiting for src bucket item count to get to: 50000 curr count: 36725
2021/12/14 17:27:09 Waiting for src bucket item count to get to: 50000 curr count: 36725
2021/12/14 17:27:14 Waiting for src bucket item count to get to: 50000 curr count: 47999
2021/12/14 17:27:19 Waiting for src bucket item count to get to: 50000 curr count: 47999
2021/12/14 17:27:29 Waiting for dst bucket item count to get to: 50000 curr count: 36636
2021/12/14 17:27:34 Waiting for dst bucket item count to get to: 50000 curr count: 36636
2021/12/14 17:27:39 Waiting for dst bucket item count to get to: 50000 curr count: 43984
2021/12/14 17:27:44 Waiting for dst bucket item count to get to: 50000 curr count: 43984
2021/12/14 17:27:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:27:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:27:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:27:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:27:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:27:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:28:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:28:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:28:14 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 17:28:14 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 17:28:14 No eventing-con process running
2021/12/14 17:28:14 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/14 17:28:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:28:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:28:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:28:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:28:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:28:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:28:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:28:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.33s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/14 17:28:43 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:28:43 Removing node(s): [] from the cluster
2021/12/14 17:28:48 Rebalance progress: 100
2021/12/14 17:28:53 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:28:53 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 17:28:53 No eventing-con process running
2021/12/14 17:28:53 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/14 17:28:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:29:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:29:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:29:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:29:12 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:29:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 17:29:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 17:29:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 17:29:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 17:29:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 17:29:33 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/14 17:29:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:29:33 Starting bucket ops to source bucket
2021/12/14 17:29:38 Rebalance progress: 12.5
2021/12/14 17:29:43 Rebalance progress: 12.5
2021/12/14 17:29:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:29:48 Waiting for src bucket item count to get to: 50000 curr count: 7875
2021/12/14 17:29:53 Waiting for src bucket item count to get to: 50000 curr count: 21395
2021/12/14 17:29:58 Waiting for src bucket item count to get to: 50000 curr count: 21395
2021/12/14 17:30:03 Waiting for src bucket item count to get to: 50000 curr count: 34785
2021/12/14 17:30:08 Waiting for src bucket item count to get to: 50000 curr count: 34785
2021/12/14 17:30:13 Waiting for src bucket item count to get to: 50000 curr count: 44133
2021/12/14 17:30:18 Waiting for src bucket item count to get to: 50000 curr count: 44133
2021/12/14 17:30:23 Waiting for src bucket item count to get to: 50000 curr count: 44133
2021/12/14 17:30:33 Waiting for dst bucket item count to get to: 50000 curr count: 28229
2021/12/14 17:30:38 Waiting for dst bucket item count to get to: 50000 curr count: 28229
2021/12/14 17:30:43 Waiting for dst bucket item count to get to: 50000 curr count: 40121
2021/12/14 17:30:48 Waiting for dst bucket item count to get to: 50000 curr count: 40121
2021/12/14 17:30:53 Waiting for dst bucket item count to get to: 50000 curr count: 49675
2021/12/14 17:30:58 Waiting for dst bucket item count to get to: 50000 curr count: 49675
2021/12/14 17:30:58 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:30:58 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:30:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:31:03 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:19 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:29 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 17:31:29 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 17:31:29 No eventing-con process running
2021/12/14 17:31:29 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/14 17:31:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:31:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:31:40 Waiting for dst bucket item count to get to: 0 curr count: 24962
2021/12/14 17:31:45 Waiting for dst bucket item count to get to: 0 curr count: 24962
2021/12/14 17:31:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:31:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:31:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:31:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (199.43s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/14 17:32:07 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:32:07 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 17:32:07 No eventing-con process running
2021/12/14 17:32:07 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/14 17:32:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:32:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:32:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:32:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:32:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:32:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:32:26 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:32:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 17:32:32 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 17:32:37 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 17:32:42 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 17:32:47 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 17:32:47 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/14 17:32:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:32:47 Starting bucket ops to source bucket
2021/12/14 17:32:48 Removing node(s): [] from the cluster
2021/12/14 17:32:53 Rebalance progress: 12.5
2021/12/14 17:32:58 Rebalance progress: 12.5
2021/12/14 17:33:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:33:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 17:33:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:33:08 Rebalance progress: 12.5
2021/12/14 17:33:13 Rebalance progress: 12.5
2021/12/14 17:33:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:33:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:33:23 Rebalance progress: 12.5
2021/12/14 17:33:28 Rebalance progress: 12.5
2021/12/14 17:33:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:33:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:33:33 Waiting for src bucket item count to get to: 50000 curr count: 47737
2021/12/14 17:33:38 Waiting for src bucket item count to get to: 50000 curr count: 47737
2021/12/14 17:33:48 Waiting for dst bucket item count to get to: 50000 curr count: 34447
2021/12/14 17:33:53 Waiting for dst bucket item count to get to: 50000 curr count: 34447
2021/12/14 17:33:58 Waiting for dst bucket item count to get to: 50000 curr count: 40625
2021/12/14 17:34:03 Waiting for dst bucket item count to get to: 50000 curr count: 40625
2021/12/14 17:34:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:34:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:34:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:34:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 17:34:34 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 17:34:34 No eventing-con process running
2021/12/14 17:34:34 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/14 17:34:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:34:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:34:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:34:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:34:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.09s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/14 17:35:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:35:07 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 17:35:07 No eventing-con process running
2021/12/14 17:35:07 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/14 17:35:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:35:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:35:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:35:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:35:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:35:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:35:26 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:35:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 17:35:32 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 17:35:37 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 17:35:42 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 17:35:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 17:35:47 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/14 17:35:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:35:47 Starting bucket ops to source bucket
2021/12/14 17:35:48 Removing node(s): [] from the cluster
2021/12/14 17:35:53 Rebalance progress: 12.5
2021/12/14 17:35:58 Rebalance progress: 12.5
2021/12/14 17:36:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:36:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:36:08 Rebalance progress: 12.5
2021/12/14 17:36:13 Rebalance progress: 12.5
2021/12/14 17:36:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:36:18 Waiting for src bucket item count to get to: 50000 curr count: 26599
2021/12/14 17:36:23 Waiting for src bucket item count to get to: 50000 curr count: 37111
2021/12/14 17:36:28 Waiting for src bucket item count to get to: 50000 curr count: 37111
2021/12/14 17:36:33 Waiting for src bucket item count to get to: 50000 curr count: 49043
2021/12/14 17:36:38 Waiting for src bucket item count to get to: 50000 curr count: 49043
2021/12/14 17:36:48 Waiting for dst bucket item count to get to: 50000 curr count: 47053
2021/12/14 17:36:53 Waiting for dst bucket item count to get to: 50000 curr count: 47053
2021/12/14 17:36:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:36:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:36:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:36:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 17:37:23 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 17:37:23 No eventing-con process running
2021/12/14 17:37:23 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/14 17:37:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:37:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:37:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:37:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:37:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:37:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:37:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.47s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/14 17:37:52 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:37:52 Removing node(s): [] from the cluster
2021/12/14 17:37:57 Rebalance progress: 100
2021/12/14 17:38:02 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:38:02 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 17:38:02 No eventing-con process running
2021/12/14 17:38:02 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/14 17:38:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:38:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:38:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:38:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:38:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:38:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:38:21 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:38:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 17:38:27 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 17:38:32 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 17:38:37 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 17:38:42 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 17:38:42 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/14 17:38:42 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:38:42 Starting bucket ops to source bucket
2021/12/14 17:38:47 Rebalance progress: 12.5
2021/12/14 17:38:52 Rebalance progress: 12.5
2021/12/14 17:38:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:38:57 Waiting for src bucket item count to get to: 50000 curr count: 9539
2021/12/14 17:39:02 Waiting for src bucket item count to get to: 50000 curr count: 21818
2021/12/14 17:39:07 Waiting for src bucket item count to get to: 50000 curr count: 21818
2021/12/14 17:39:12 Waiting for src bucket item count to get to: 50000 curr count: 33084
2021/12/14 17:39:17 Waiting for src bucket item count to get to: 50000 curr count: 33084
2021/12/14 17:39:22 Waiting for src bucket item count to get to: 50000 curr count: 43570
2021/12/14 17:39:27 Waiting for src bucket item count to get to: 50000 curr count: 43570
2021/12/14 17:39:32 Waiting for src bucket item count to get to: 50000 curr count: 43570
2021/12/14 17:39:42 Waiting for dst bucket item count to get to: 50000 curr count: 39992
2021/12/14 17:39:47 Waiting for dst bucket item count to get to: 50000 curr count: 39992
2021/12/14 17:39:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:39:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:39:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:39:52 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:39:57 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:40:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:40:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:40:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:40:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 17:40:18 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 17:40:18 No eventing-con process running
2021/12/14 17:40:18 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/14 17:40:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:40:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:40:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:40:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:40:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:40:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (169.41s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/14 17:40:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:40:46 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 17:40:46 No eventing-con process running
2021/12/14 17:40:46 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/14 17:40:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:40:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:40:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:41:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:41:05 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:41:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 17:41:11 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 17:41:16 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 17:41:21 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 17:41:26 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 17:41:26 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/14 17:41:26 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 17:41:26 Starting bucket ops to source bucket
2021/12/14 17:41:27 Removing node(s): [] from the cluster
2021/12/14 17:41:32 Rebalance progress: 12.5
2021/12/14 17:41:37 Rebalance progress: 12.5
2021/12/14 17:41:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:41:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 17:41:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:41:47 Rebalance progress: 12.5
2021/12/14 17:41:52 Rebalance progress: 12.5
2021/12/14 17:41:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:41:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:41:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:42:02 Rebalance progress: 12.5
2021/12/14 17:42:07 Rebalance progress: 12.5
2021/12/14 17:42:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:42:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:42:12 Waiting for src bucket item count to get to: 50000 curr count: 49535
2021/12/14 17:42:17 Waiting for src bucket item count to get to: 50000 curr count: 49535
2021/12/14 17:42:27 Waiting for dst bucket item count to get to: 50000 curr count: 42597
2021/12/14 17:42:32 Waiting for dst bucket item count to get to: 50000 curr count: 42597
2021/12/14 17:42:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:42:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:42:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:42:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:42:42 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:42:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:42:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:42:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:43:03 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 17:43:03 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 17:43:03 No eventing-con process running
2021/12/14 17:43:03 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/14 17:43:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:43:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:43:14 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:43:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:43:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:43:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:43:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:43:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.99s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/14 17:43:36 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:43:36 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 17:43:36 No eventing-con process running
2021/12/14 17:43:36 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/14 17:43:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:43:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:43:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:43:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:43:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:43:55 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:43:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 17:44:01 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 17:44:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 17:44:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 17:44:16 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 17:44:16 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/14 17:44:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:44:16 Starting bucket ops to source bucket
2021/12/14 17:44:16 Removing node(s): [] from the cluster
2021/12/14 17:44:21 Rebalance progress: 12.5
2021/12/14 17:44:26 Rebalance progress: 12.5
2021/12/14 17:44:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:44:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:44:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:44:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:44:36 Rebalance progress: 12.5
2021/12/14 17:44:41 Rebalance progress: 12.5
2021/12/14 17:44:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:44:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:44:46 Waiting for src bucket item count to get to: 50000 curr count: 29108
2021/12/14 17:44:51 Waiting for src bucket item count to get to: 50000 curr count: 39680
2021/12/14 17:44:56 Waiting for src bucket item count to get to: 50000 curr count: 39680
2021/12/14 17:45:06 Waiting for dst bucket item count to get to: 50000 curr count: 34572
2021/12/14 17:45:11 Waiting for dst bucket item count to get to: 50000 curr count: 34572
2021/12/14 17:45:16 Waiting for dst bucket item count to get to: 50000 curr count: 42399
2021/12/14 17:45:21 Waiting for dst bucket item count to get to: 50000 curr count: 42399
2021/12/14 17:45:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:45:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:45:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:45:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:32 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:37 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 17:45:52 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 17:45:52 No eventing-con process running
2021/12/14 17:45:52 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/14 17:45:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:46:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:46:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:46:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:46:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:46:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:46:21 Waiting for dst bucket item count to get to: 0 curr count: 28119
2021/12/14 17:46:26 Waiting for dst bucket item count to get to: 0 curr count: 28119
2021/12/14 17:46:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (179.26s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/14 17:46:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:46:31 Removing node(s): [] from the cluster
2021/12/14 17:46:36 Rebalance progress: 100
2021/12/14 17:46:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:46:41 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 17:46:41 No eventing-con process running
2021/12/14 17:46:41 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/14 17:46:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:46:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:46:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:46:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:46:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:46:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:46:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:46:59 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:47:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 17:47:05 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 17:47:10 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 17:47:15 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 17:47:20 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 17:47:20 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/14 17:47:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:47:20 Starting bucket ops to source bucket
2021/12/14 17:47:26 Rebalance progress: 12.5
2021/12/14 17:47:31 Rebalance progress: 12.5
2021/12/14 17:47:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:47:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:47:36 Waiting for src bucket item count to get to: 50000 curr count: 11761
2021/12/14 17:47:41 Waiting for src bucket item count to get to: 50000 curr count: 24015
2021/12/14 17:47:46 Waiting for src bucket item count to get to: 50000 curr count: 24015
2021/12/14 17:47:51 Waiting for src bucket item count to get to: 50000 curr count: 33877
2021/12/14 17:47:56 Waiting for src bucket item count to get to: 50000 curr count: 33877
2021/12/14 17:48:01 Waiting for src bucket item count to get to: 50000 curr count: 33877
2021/12/14 17:48:06 Waiting for src bucket item count to get to: 50000 curr count: 44875
2021/12/14 17:48:11 Waiting for src bucket item count to get to: 50000 curr count: 44875
2021/12/14 17:48:21 Waiting for dst bucket item count to get to: 50000 curr count: 40614
2021/12/14 17:48:26 Waiting for dst bucket item count to get to: 50000 curr count: 40614
2021/12/14 17:48:26 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:48:26 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:48:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:48:31 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:48:36 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:48:41 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:48:46 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:48:51 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 17:48:51 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 17:48:51 No eventing-con process running
2021/12/14 17:48:51 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/14 17:48:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:49:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:49:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:49:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:49:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (169.22s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/14 17:49:25 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:49:25 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 17:49:25 No eventing-con process running
2021/12/14 17:49:25 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/14 17:49:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:49:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:49:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:49:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:49:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:49:43 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:49:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 17:49:49 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 17:49:54 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 17:49:59 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 17:50:05 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 17:50:05 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/14 17:50:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:50:05 Starting bucket ops to source bucket
2021/12/14 17:50:05 Removing node(s): [] from the cluster
2021/12/14 17:50:10 Rebalance progress: 12.5
2021/12/14 17:50:15 Rebalance progress: 12.5
2021/12/14 17:50:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:50:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 17:50:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:50:26 Rebalance progress: 12.5
2021/12/14 17:50:31 Rebalance progress: 12.5
2021/12/14 17:50:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:50:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:50:41 Rebalance progress: 12.5
2021/12/14 17:50:46 Rebalance progress: 12.5
2021/12/14 17:50:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:50:56 Waiting for dst bucket item count to get to: 50000 curr count: 33255
2021/12/14 17:51:01 Waiting for dst bucket item count to get to: 50000 curr count: 33255
2021/12/14 17:51:06 Waiting for dst bucket item count to get to: 50000 curr count: 41687
2021/12/14 17:51:11 Waiting for dst bucket item count to get to: 50000 curr count: 41687
2021/12/14 17:51:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 17:51:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 17:51:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:51:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:21 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:26 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 17:51:41 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 17:51:41 No eventing-con process running
2021/12/14 17:51:41 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/14 17:51:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:51:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:51:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:51:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 17:51:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:52:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:52:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.09s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/14 17:52:15 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:52:15 App: bucket_op_on_update got un-deployed
2021/12/14 17:52:15 No eventing-con process running
2021/12/14 17:52:15 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 17:52:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:52:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:52:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:52:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:52:33 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:52:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 17:52:40 Starting bucket ops to source bucket
2021/12/14 17:52:45 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:52:50 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:52:55 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:52:55 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:52:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:52:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:52:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 17:53:00 Rebalance progress: 12.5
2021/12/14 17:53:05 Rebalance progress: 12.5
2021/12/14 17:53:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:53:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 17:53:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 17:53:16 Rebalance progress: 12.5
2021/12/14 17:53:21 Rebalance progress: 12.5
2021/12/14 17:53:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:53:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 17:53:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 17:53:31 Rebalance progress: 12.5
2021/12/14 17:53:36 Rebalance progress: 12.5
2021/12/14 17:53:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:53:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 17:53:47 Rebalance progress: 12.5
2021/12/14 17:53:52 Rebalance progress: 12.5
2021/12/14 17:53:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:53:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:53:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 17:54:02 Rebalance progress: 12.5
2021/12/14 17:54:07 Rebalance progress: 12.5
2021/12/14 17:54:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:54:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:54:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 17:54:17 Rebalance progress: 12.5
2021/12/14 17:54:22 Rebalance progress: 12.5
2021/12/14 17:54:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:54:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:54:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:54:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:54:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:54:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:54:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:54:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:54:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 17:54:52 eventingSetup.go:46: No eventing-con process running
2021/12/14 17:54:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 17:54:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:54:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:55:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10707
2021/12/14 17:55:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10707
2021/12/14 17:55:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:55:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:55:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:55:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7674
2021/12/14 17:55:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7674
2021/12/14 17:55:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.86s)
=== RUN   TestKVForceFailover
2021/12/14 17:55:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:55:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 17:55:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 17:55:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 17:55:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:55:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:55:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:55:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:55:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:55:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:55:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:55:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 17:56:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:56:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:56:00 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:56:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 17:56:01 Starting bucket ops to source bucket
2021/12/14 17:56:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:56:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:56:16 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:56:21 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:56:21 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:56:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:56:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:56:21 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 17:56:41 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 17:56:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 17:56:47 Rebalance progress: 12.5
2021/12/14 17:56:52 Rebalance progress: 12.5
2021/12/14 17:56:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:56:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:56:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:56:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 17:57:02 Rebalance progress: 12.5
2021/12/14 17:57:07 Rebalance progress: 12.5
2021/12/14 17:57:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:57:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:57:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 17:57:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:57:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 17:57:42 eventingSetup.go:46: No eventing-con process running
2021/12/14 17:57:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 17:57:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:57:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:57:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7091
2021/12/14 17:57:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7091
2021/12/14 17:57:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:58:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:58:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:58:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6162
2021/12/14 17:58:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6162
2021/12/14 17:58:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (170.18s)
=== RUN   TestKVHardFailoverRecovery
2021/12/14 17:58:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 17:58:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 17:58:26 eventingSetup.go:46: No eventing-con process running
2021/12/14 17:58:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 17:58:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:58:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:58:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 17:58:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:58:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:58:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:58:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 17:58:44 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 17:58:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 17:58:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 17:58:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:59:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:59:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 17:59:06 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:59:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 17:59:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 17:59:11 Rebalance progress: 12.5
2021/12/14 17:59:16 Rebalance progress: 12.5
2021/12/14 17:59:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 17:59:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 17:59:21 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 18:00:21 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 18:00:26 Rebalance progress: 12.5
2021/12/14 18:00:31 Rebalance progress: 12.5
2021/12/14 18:00:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:00:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:00:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:00:41 Rebalance progress: 12.5
2021/12/14 18:00:46 Rebalance progress: 12.5
2021/12/14 18:00:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:00:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:00:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:01:22 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:01:22 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:01:22 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:01:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:01:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:01:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12066
2021/12/14 18:01:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12066
2021/12/14 18:01:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:01:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:01:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:01:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4149
2021/12/14 18:01:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4149
2021/12/14 18:01:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (219.39s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/14 18:02:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:02:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:02:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:02:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:02:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:02:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:02:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:02:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:02:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:02:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:02:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:02:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:02:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 18:02:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:02:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:02:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:02:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:02:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:02:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:02:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:02:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:02:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:02:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 18:02:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:02:51 Rebalance progress: 12.5
2021/12/14 18:02:56 Rebalance progress: 12.5
2021/12/14 18:03:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:03:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:03:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:03:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 18:03:07 Rebalance progress: 12.5
2021/12/14 18:03:12 Rebalance progress: 12.5
2021/12/14 18:03:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:03:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:03:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:03:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:03:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:03:47 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:03:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:03:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:03:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:03:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4651
2021/12/14 18:03:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:04:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:04:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:04:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2932
2021/12/14 18:04:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2932
2021/12/14 18:04:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (140.32s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/14 18:04:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:04:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:04:26 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:04:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:04:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:04:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:04:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:04:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:04:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:04:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:04:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:04:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:04:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 18:04:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:04:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:04:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:05:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:05:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:05:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:05:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:05:06 Rebalance progress: 12.5
2021/12/14 18:05:11 Rebalance progress: 12.5
2021/12/14 18:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:05:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:05:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:05:21 Rebalance progress: 12.5
2021/12/14 18:05:26 Rebalance progress: 12.5
2021/12/14 18:05:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:05:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 18:05:36 Rebalance progress: 12.5
2021/12/14 18:05:41 Rebalance progress: 12.5
2021/12/14 18:05:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:05:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:05:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:05:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:05:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:06:16 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:06:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:06:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:06:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:06:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6057
2021/12/14 18:06:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:06:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:06:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:06:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 753
2021/12/14 18:06:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 753
2021/12/14 18:06:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:06:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:06:50 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:06:50 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:06:50 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:06:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:06:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:06:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:06:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.06s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/14 18:07:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:07:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:07:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:07:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:07:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:07:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:07:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:07:32 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 18:07:32 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:07:32 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 18:07:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:07:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:46 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:07:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:07:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:07:47 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:07:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 18:07:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:07:58 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:08:03 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:08:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:08:08 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 18:08:13 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:08:18 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:08:23 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:08:23 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:08:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:08:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:08:29 Rebalance progress: 12.5
2021/12/14 18:08:34 Rebalance progress: 12.5
2021/12/14 18:08:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:08:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:08:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:08:45 Rebalance progress: 12.5
2021/12/14 18:08:50 Rebalance progress: 12.5
2021/12/14 18:08:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:08:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:08:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 18:08:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:09:00 Rebalance progress: 12.5
2021/12/14 18:09:05 Rebalance progress: 12.5
2021/12/14 18:09:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:09:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:09:16 Rebalance progress: 12.5
2021/12/14 18:09:21 Rebalance progress: 12.5
2021/12/14 18:09:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:09:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 18:09:31 Rebalance progress: 12.5
2021/12/14 18:09:36 Rebalance progress: 12.5
2021/12/14 18:09:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:09:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 18:09:46 Rebalance progress: 12.5
2021/12/14 18:09:51 Rebalance progress: 12.5
2021/12/14 18:09:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:09:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:10:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:10:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:10:31 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:10:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:10:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:10:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:10:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12259
2021/12/14 18:10:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:10:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:10:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:10:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12259
2021/12/14 18:10:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:10:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:11:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:29 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:11:29 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 18:11:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:11:29 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 18:11:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:11:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:11:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:11:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:11:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:11:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (278.96s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/14 18:11:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:11:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:11:53 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:11:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:11:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:11:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:11:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:12:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:12:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:12:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:12:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:12:11 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 18:12:11 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:12:11 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 18:12:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:12:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:12:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:12:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:12:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:12:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:12:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:12:25 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:12:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:12:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:12:26 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:12:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 18:12:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:12:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:12:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:12:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:12:47 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 18:12:52 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:12:57 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:13:02 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 18:13:02 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:13:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:13:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:13:08 Rebalance progress: 12.5
2021/12/14 18:13:13 Rebalance progress: 12.5
2021/12/14 18:13:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:13:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:13:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:13:23 Rebalance progress: 12.5
2021/12/14 18:13:28 Rebalance progress: 12.5
2021/12/14 18:13:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:13:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 18:13:38 Rebalance progress: 12.5
2021/12/14 18:13:43 Rebalance progress: 12.5
2021/12/14 18:13:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:13:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:13:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:13:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:13:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:14:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:14:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:14:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:14:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:14:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:14:19 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:14:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:14:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:14:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:14:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5255
2021/12/14 18:14:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5255
2021/12/14 18:14:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:14:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:14:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:14:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7589
2021/12/14 18:14:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:14:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:14:54 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:14:59 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:15:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:15:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:15:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 18:15:14 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 18:15:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:15:14 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 18:15:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:15:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:15:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:15:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:15:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:15:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (225.23s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 18:15:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:15:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:15:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:15:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:15:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:15:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:15:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:15:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:15:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:15:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:15:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:15:56 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:15:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 18:16:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:16:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:16:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:16:18 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:16:18 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:16:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:16:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:16:38 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:16:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:16:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:16:54 Rebalance progress: 12.5
2021/12/14 18:16:59 Rebalance progress: 12.5
2021/12/14 18:17:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:17:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:17:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:17:09 Rebalance progress: 12.5
2021/12/14 18:17:14 Rebalance progress: 12.5
2021/12/14 18:17:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:17:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 18:17:24 Rebalance progress: 12.5
2021/12/14 18:17:29 Rebalance progress: 12.5
2021/12/14 18:17:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:17:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:17:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:17:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:17:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:17:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:17:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:17:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:18:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:18:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:18:04 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:18:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:18:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:18:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:18:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9141
2021/12/14 18:18:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:18:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:18:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:18:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9141
2021/12/14 18:18:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (180.22s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/14 18:18:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:18:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:18:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:18:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 18:18:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:18:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:18:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:18:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:18:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:18:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:18:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:18:57 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:18:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 18:19:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 18:19:08 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:19:13 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:19:18 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:19:18 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:19:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:19:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:19:23 Rebalance progress: 12.5
2021/12/14 18:19:28 Rebalance progress: 12.5
2021/12/14 18:19:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:19:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:19:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:19:48 Rebalance progress: 12.5
2021/12/14 18:19:53 Rebalance progress: 12.5
2021/12/14 18:19:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:19:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:19:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 18:19:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 18:20:04 Rebalance progress: 12.5
2021/12/14 18:20:09 Rebalance progress: 12.5
2021/12/14 18:20:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:20:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:20:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:20:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 18:20:19 Rebalance progress: 12.5
2021/12/14 18:20:24 Rebalance progress: 12.5
2021/12/14 18:20:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:20:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:20:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:20:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:20:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:20:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:20:59 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:20:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:21:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:21:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:21:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8603
2021/12/14 18:21:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8603
2021/12/14 18:21:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:21:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:21:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:21:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8603
2021/12/14 18:21:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (180.08s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/14 18:21:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 18:21:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 18:21:39 Rebalance progress: 100
2021/12/14 18:21:39 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:22:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:22:09 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 18:22:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 18:22:15 Removing node(s): [] from the cluster
2021/12/14 18:22:15 Starting bucket ops to source bucket
2021/12/14 18:22:20 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/14 18:22:25 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:22:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 18:22:31 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 18:22:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 18:22:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:22:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:22:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:22:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:22:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:23:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 18:23:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 18:23:01 eventingSetup.go:46: No eventing-con process running
2021/12/14 18:23:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 18:23:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:23:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:23:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/14 18:23:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/14 18:23:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 18:23:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 18:23:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 18:23:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (116.36s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12170.407s

Rebalance tests took 3 hours, 22 minutes and 53 seconds.

Testing Succeeded


Version: versions-14.12.2021-10.37.cfg
Build Log: make-14.12.2021-10.37.log
Server Log: logs-14.12.2021-10.37.tar.gz

Finished