Building

Started building at 2021/12/13 10:19:36
Calculating base
Updating mirror
Basing run on 7.1.0-1896 4ac5b3bbd
Updating tree for run 13.12.2021-10.19
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: none
kv_engine is at 2499d82, changes since last good build: none
ns_server is at 5bd87dc, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2f97349, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 5a6bde2e] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 9f7f3cfa] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 89387207] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable a9b4e673] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 540cf69f] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable a4c0b616] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 02de9558] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable 5510958d] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable fa73e3f] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/13 11:15:22
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/13 11:16:53 Started functional
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:35024->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:35024->127.0.0.1:9000: read: connection reset by peer
2021/12/13 11:17:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 11:17:09 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/13 11:17:09 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 11:17:15 Removing node(s): [] from the cluster
2021/12/13 11:17:20 Rebalance progress: 0
2021/12/13 11:17:25 Rebalance progress: 0.4340277777777777
2021/12/13 11:17:30 Rebalance progress: 5.081741898148148
2021/12/13 11:17:35 Rebalance progress: 9.25925925925926
2021/12/13 11:17:40 Rebalance progress: 13.0931712962963
2021/12/13 11:17:45 Rebalance progress: 17.77705439814815
2021/12/13 11:17:50 Rebalance progress: 21.01417824074074
2021/12/13 11:17:55 Rebalance progress: 25.58955439814815
2021/12/13 11:18:00 Rebalance progress: 100
2021/12/13 11:18:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:18:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:18:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:18:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:19:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:19:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:19:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:19:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:20:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:20:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:20:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:20:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:21:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:21:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:21:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/13 11:21:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/13 11:22:05 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:22:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:22:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 11:22:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 11:22:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 11:22:27 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 11:22:27 App: TestAdvancedGetOps got deployed
2021/12/13 11:22:27 Starting bucket ops to source bucket
2021/12/13 11:22:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 11:22:37 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 11:22:37 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 11:22:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:22:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:22:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:22:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 11:22:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 11:22:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 11:22:57 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 11:22:57 App: TestAdvancedGetOps got un-deployed
2021/12/13 11:22:57 No eventing-con process running
2021/12/13 11:22:57 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/13 11:22:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:23:09 Waiting for dst bucket item count to get to: 0 curr count: 92
2021/12/13 11:23:14 Waiting for dst bucket item count to get to: 0 curr count: 42
2021/12/13 11:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:23:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:23:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:23:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.15s)
=== RUN   TestAdvancedInsertOps
2021/12/13 11:23:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:23:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:23:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:23:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:23:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:23:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:23:47 App: TestAdvancedInsertOps got deployed
2021/12/13 11:23:47 Starting bucket ops to source bucket
2021/12/13 11:23:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 11:23:58 Waiting for dst bucket item count to get to: 1024 curr count: 801
2021/12/13 11:24:03 Waiting for dst bucket item count to get to: 1024 curr count: 801
2021/12/13 11:24:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 11:24:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:24:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:24:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:24:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:24:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:24:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:24:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:24:23 App: TestAdvancedInsertOps got un-deployed
2021/12/13 11:24:23 No eventing-con process running
2021/12/13 11:24:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/13 11:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:24:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:24:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:24:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:24:46 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:24:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:24:46 Testing insert operation on source bucket
2021/12/13 11:24:46 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:24:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:24:52 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:24:57 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:25:02 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:25:07 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 11:25:07 App: TestAdvancedInsertOps got deployed
2021/12/13 11:25:07 Starting bucket ops to source bucket
2021/12/13 11:25:13 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 11:25:18 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/13 11:25:23 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/13 11:25:23 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 11:25:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:25:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:25:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:25:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:25:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:25:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:25:44 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 11:25:44 App: TestAdvancedInsertOps got un-deployed
2021/12/13 11:25:44 No eventing-con process running
2021/12/13 11:25:44 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/13 11:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:25:55 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/13 11:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:25:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:26:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.58s)
=== RUN   TestAdvancedUpsertOps
2021/12/13 11:26:02 Starting bucket ops to source bucket
2021/12/13 11:26:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:26:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:26:09 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:26:14 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:26:19 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:26:24 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:26:24 App: TestAdvancedUpsertOps got deployed
2021/12/13 11:26:24 Starting bucket ops to source bucket
2021/12/13 11:26:30 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:26:35 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 11:26:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:26:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:26:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:26:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:26:40 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:26:45 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:26:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:26:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:26:55 App: TestAdvancedUpsertOps got un-deployed
2021/12/13 11:26:55 No eventing-con process running
2021/12/13 11:26:55 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/13 11:26:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:27:07 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:27:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:27:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:27:14 Testing upsert operation on source bucket
2021/12/13 11:27:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:27:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:27:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:27:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:27:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:27:35 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 11:27:35 App: TestAdvancedUpsertOps got deployed
2021/12/13 11:27:35 Starting bucket ops to source bucket
2021/12/13 11:27:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:27:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:27:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:27:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:27:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:27:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:27:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:27:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 11:27:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/13 11:27:56 No eventing-con process running
2021/12/13 11:27:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/13 11:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:28:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:28:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:28:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.03s)
=== RUN   TestAdvancedReplaceOps
2021/12/13 11:28:10 Starting bucket ops to source bucket
2021/12/13 11:28:11 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:28:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:28:17 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:28:22 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:28:27 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:28:32 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:28:32 App: TestAdvancedReplaceOps got deployed
2021/12/13 11:28:32 Starting bucket ops to source bucket
2021/12/13 11:28:38 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:28:43 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:28:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:28:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:28:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:28:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:28:48 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:28:53 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:28:58 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:29:03 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:29:03 App: TestAdvancedReplaceOps got un-deployed
2021/12/13 11:29:03 No eventing-con process running
2021/12/13 11:29:03 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/13 11:29:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:29:15 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 11:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:29:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:29:22 Testing upsert operation on source bucket
2021/12/13 11:29:22 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:29:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:29:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:29:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:29:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:29:43 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 11:29:43 App: TestAdvancedReplaceOps got deployed
2021/12/13 11:29:43 Starting bucket ops to source bucket
2021/12/13 11:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:29:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:29:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:29:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:29:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:29:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:29:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:30:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 11:30:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/13 11:30:04 No eventing-con process running
2021/12/13 11:30:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/13 11:30:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:30:16 Waiting for dst bucket item count to get to: 0 curr count: 336
2021/12/13 11:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:30:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:30:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (132.66s)
=== RUN   TestAdvancedDeleteOps
2021/12/13 11:30:23 Starting bucket ops to source bucket
2021/12/13 11:30:23 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:30:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:30:29 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:30:34 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:30:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:30:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:30:44 App: TestAdvancedDeleteOps got deployed
2021/12/13 11:30:44 Starting bucket ops to source bucket
2021/12/13 11:30:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:30:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:30:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:30:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:30:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:30:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:30:59 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:31:04 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:31:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:31:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:31:14 App: TestAdvancedDeleteOps got un-deployed
2021/12/13 11:31:14 No eventing-con process running
2021/12/13 11:31:14 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/13 11:31:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:31:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:31:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:31:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:31:33 Testing delete operation on source bucket
2021/12/13 11:31:33 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:31:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/13 11:31:39 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:31:44 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:31:49 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:31:54 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 11:31:54 App: TestAdvancedDeleteOps got deployed
2021/12/13 11:31:54 Starting bucket ops to source bucket
2021/12/13 11:31:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:31:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:31:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:31:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:31:59 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:32:04 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:32:09 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:32:14 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 11:32:14 App: TestAdvancedDeleteOps got un-deployed
2021/12/13 11:32:14 No eventing-con process running
2021/12/13 11:32:14 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/13 11:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:32:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:32:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:32:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.76s)
=== RUN   TestEnoentAdvancedGet
2021/12/13 11:32:29 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:32:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/13 11:32:35 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:32:40 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:32:45 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:32:50 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:32:50 App: TestEnoentAdvancedGet got deployed
2021/12/13 11:32:50 Starting bucket ops to source bucket
2021/12/13 11:32:55 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 11:33:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 11:33:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 11:33:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 11:33:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:33:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:33:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:33:10 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:33:15 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:33:20 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:33:25 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:33:25 App: TestEnoentAdvancedGet got un-deployed
2021/12/13 11:33:25 No eventing-con process running
2021/12/13 11:33:25 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/13 11:33:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:33:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:33:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:33:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:33:48 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/13 11:33:54 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/13 11:33:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:33:54 Testing get enoent operation on source bucket
2021/12/13 11:33:54 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:33:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/13 11:34:00 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:34:05 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:34:10 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:34:15 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 11:34:15 App: TestEnoentAdvancedGet got deployed
2021/12/13 11:34:15 Starting bucket ops to source bucket
2021/12/13 11:34:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 11:34:25 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 11:34:25 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 11:34:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:34:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:34:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:34:30 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:34:35 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:34:40 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:34:45 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 11:34:45 App: TestEnoentAdvancedGet got un-deployed
2021/12/13 11:34:45 No eventing-con process running
2021/12/13 11:34:45 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/13 11:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:34:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/13 11:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:34:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:35:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.99s)
=== RUN   TestEnoentAdvancedReplace
2021/12/13 11:35:04 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:35:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:35:10 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:35:15 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:35:20 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:35:25 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:35:25 App: TestEnoentAdvancedReplace got deployed
2021/12/13 11:35:25 Starting bucket ops to source bucket
2021/12/13 11:35:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 11:35:35 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 11:35:35 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 11:35:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:35:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:35:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:35:40 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:35:45 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:35:50 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:35:55 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:35:55 App: TestEnoentAdvancedReplace got un-deployed
2021/12/13 11:35:55 No eventing-con process running
2021/12/13 11:35:55 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/13 11:35:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:36:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:36:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:36:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:36:18 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/13 11:36:23 Waiting for dst bucket item count to get to: 0 curr count: 34
2021/12/13 11:36:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:36:23 Testing get enoent operation on source bucket
2021/12/13 11:36:23 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:36:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:36:30 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:36:35 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:36:40 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:36:45 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 11:36:45 App: TestEnoentAdvancedReplace got deployed
2021/12/13 11:36:45 Starting bucket ops to source bucket
2021/12/13 11:36:50 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 11:36:55 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 11:36:55 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 11:36:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:36:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:36:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:37:00 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:37:05 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:37:10 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:37:15 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 11:37:15 App: TestEnoentAdvancedReplace got un-deployed
2021/12/13 11:37:15 No eventing-con process running
2021/12/13 11:37:15 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/13 11:37:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:37:27 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/13 11:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:37:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:37:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.08s)
=== RUN   TestEnoentAdvancedDelete
2021/12/13 11:37:34 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:37:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 11:37:40 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:37:45 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:37:50 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:37:55 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:37:55 App: TestEnoentAdvancedDelete got deployed
2021/12/13 11:37:55 Starting bucket ops to source bucket
2021/12/13 11:38:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 11:38:05 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 11:38:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 11:38:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:38:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:38:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:38:10 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:38:15 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:38:20 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:38:25 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:38:25 App: TestEnoentAdvancedDelete got un-deployed
2021/12/13 11:38:25 No eventing-con process running
2021/12/13 11:38:25 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/13 11:38:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:38:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:38:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:38:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:38:49 Waiting for dst bucket item count to get to: 0 curr count: 86
2021/12/13 11:38:54 Waiting for dst bucket item count to get to: 0 curr count: 36
2021/12/13 11:38:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:38:54 Testing get enoent operation on source bucket
2021/12/13 11:38:54 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:38:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 11:39:00 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:39:05 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:39:10 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:39:15 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 11:39:15 App: TestEnoentAdvancedDelete got deployed
2021/12/13 11:39:15 Starting bucket ops to source bucket
2021/12/13 11:39:20 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 11:39:25 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 11:39:25 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 11:39:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:39:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:39:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:39:30 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:39:35 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:39:40 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:39:45 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 11:39:45 App: TestEnoentAdvancedDelete got un-deployed
2021/12/13 11:39:45 No eventing-con process running
2021/12/13 11:39:45 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/13 11:39:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:39:57 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/13 11:39:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:39:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:40:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.97s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/13 11:40:04 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:40:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/13 11:40:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:40:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:40:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:40:25 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:40:25 App: TestAdvancedInsertKeyExist got deployed
2021/12/13 11:40:25 Starting bucket ops to source bucket
2021/12/13 11:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:40:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:40:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:40:30 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:40:35 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:40:40 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:40:45 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:40:45 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/13 11:40:45 No eventing-con process running
2021/12/13 11:40:45 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/13 11:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:40:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:40:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:40:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:40:59 Pumping document in the destination bucket
2021/12/13 11:40:59 Starting bucket ops to source bucket
2021/12/13 11:40:59 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:41:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/13 11:41:05 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:41:10 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:41:15 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:41:20 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 11:41:20 App: TestAdvancedInsertKeyExist got deployed
2021/12/13 11:41:20 Starting bucket ops to source bucket
2021/12/13 11:41:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:41:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:41:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:41:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:41:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:41:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:41:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:41:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:41:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:41:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 11:41:51 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/13 11:41:51 No eventing-con process running
2021/12/13 11:41:51 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/13 11:41:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:42:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:42:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:42:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:42:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.63s)
=== RUN   TestExipryGet
2021/12/13 11:42:14 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:42:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/13 11:42:21 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:42:26 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:42:31 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:42:36 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:42:36 App: TestExipryGet got deployed
2021/12/13 11:42:36 Starting bucket ops to source bucket
2021/12/13 11:43:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:43:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:43:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:43:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:43:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:43:26 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:43:31 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:43:36 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:43:41 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:43:41 App: TestExipryGet got un-deployed
2021/12/13 11:43:41 No eventing-con process running
2021/12/13 11:43:41 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/13 11:43:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:43:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:43:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:43:55 Pumping document in the destination bucket for expiry get
2021/12/13 11:43:55 Starting bucket ops to source bucket
2021/12/13 11:43:55 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:43:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/13 11:44:01 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:44:06 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:44:11 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:44:16 Waiting for app: TestExipryGet to get deployed
2021/12/13 11:44:16 App: TestExipryGet got deployed
2021/12/13 11:44:16 Starting bucket ops to source bucket
2021/12/13 11:45:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:45:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:45:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:45:07 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:45:12 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:45:17 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:45:22 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 11:45:22 App: TestExipryGet got un-deployed
2021/12/13 11:45:22 No eventing-con process running
2021/12/13 11:45:22 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/13 11:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:45:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:45:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:45:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (210.99s)
=== RUN   TestExipryInsert
2021/12/13 11:45:45 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:45:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/13 11:45:51 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:45:56 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:46:01 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:46:06 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:46:06 App: TestExipryInsert got deployed
2021/12/13 11:46:06 Starting bucket ops to source bucket
2021/12/13 11:46:52 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/13 11:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:46:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:46:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:46:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:46:57 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:47:02 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:47:07 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:47:12 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:47:12 App: TestExipryInsert got un-deployed
2021/12/13 11:47:12 No eventing-con process running
2021/12/13 11:47:12 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/13 11:47:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:47:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:47:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:47:26 Testing on the destination bucket for expiry insert
2021/12/13 11:47:26 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:47:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/13 11:47:32 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:47:37 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:47:42 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:47:47 Waiting for app: TestExipryInsert to get deployed
2021/12/13 11:47:47 App: TestExipryInsert got deployed
2021/12/13 11:47:47 Starting bucket ops to source bucket
2021/12/13 11:48:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:48:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:48:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:48:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:48:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:48:37 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:48:42 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:48:47 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:48:52 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 11:48:52 App: TestExipryInsert got un-deployed
2021/12/13 11:48:52 No eventing-con process running
2021/12/13 11:48:52 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/13 11:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:49:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:49:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:49:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.57s)
=== RUN   TestExipryReplace
2021/12/13 11:49:16 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:49:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:49:22 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:49:27 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:49:32 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:49:37 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:49:37 App: TestExipryReplace got deployed
2021/12/13 11:49:37 Starting bucket ops to source bucket
2021/12/13 11:50:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:50:27 Waiting for dst bucket item count to get to: 0 curr count: 38
2021/12/13 11:50:32 Waiting for dst bucket item count to get to: 0 curr count: 38
2021/12/13 11:50:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:50:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:50:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:50:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:50:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:50:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:50:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:50:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:50:52 App: TestExipryReplace got un-deployed
2021/12/13 11:50:52 No eventing-con process running
2021/12/13 11:50:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/13 11:50:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:50:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:51:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:51:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:51:06 Testing on the destination bucket for expiry insert
2021/12/13 11:51:06 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:51:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:51:13 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:51:18 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:51:23 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:51:28 Waiting for app: TestExipryReplace to get deployed
2021/12/13 11:51:28 App: TestExipryReplace got deployed
2021/12/13 11:51:28 Starting bucket ops to source bucket
2021/12/13 11:52:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:52:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:52:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:52:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:52:18 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:52:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:52:23 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:52:28 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:52:33 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:52:38 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 11:52:38 App: TestExipryReplace got un-deployed
2021/12/13 11:52:38 No eventing-con process running
2021/12/13 11:52:38 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/13 11:52:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:52:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:52:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:52:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:53:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (225.69s)
=== RUN   TestExipryUpsert
2021/12/13 11:53:01 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:53:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/13 11:53:08 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:53:13 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:53:18 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:53:23 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:53:23 App: TestExipryUpsert got deployed
2021/12/13 11:53:23 Starting bucket ops to source bucket
2021/12/13 11:54:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:54:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:54:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:54:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:54:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:54:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:54:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:54:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:54:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:54:33 App: TestExipryUpsert got un-deployed
2021/12/13 11:54:33 No eventing-con process running
2021/12/13 11:54:33 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/13 11:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:54:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:54:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:54:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:54:47 Testing on the destination bucket for expiry insert
2021/12/13 11:54:47 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:54:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/13 11:54:53 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:54:58 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:55:03 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:55:08 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 11:55:08 App: TestExipryUpsert got deployed
2021/12/13 11:55:08 Starting bucket ops to source bucket
2021/12/13 11:55:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:55:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:55:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:55:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:55:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:55:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:56:03 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:56:08 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:56:13 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:56:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 11:56:18 App: TestExipryUpsert got un-deployed
2021/12/13 11:56:18 No eventing-con process running
2021/12/13 11:56:19 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/13 11:56:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:56:30 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/13 11:56:35 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/13 11:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:56:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:56:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.55s)
=== RUN   TestCasReplace
2021/12/13 11:56:42 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:56:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:56:48 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:56:53 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:56:58 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:57:03 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:57:03 App: TestCasReplace got deployed
2021/12/13 11:57:03 Starting bucket ops to source bucket
2021/12/13 11:57:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:57:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:57:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:57:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:57:09 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:57:14 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:57:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:57:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:57:24 App: TestCasReplace got un-deployed
2021/12/13 11:57:24 No eventing-con process running
2021/12/13 11:57:24 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/13 11:57:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:57:35 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/13 11:57:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:57:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:57:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:57:42 Testing on the destination bucket for upsert with cas
2021/12/13 11:57:42 Starting bucket ops to source bucket
2021/12/13 11:57:42 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:57:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 11:57:48 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:57:53 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:57:58 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:58:03 Waiting for app: TestCasReplace to get deployed
2021/12/13 11:58:03 App: TestCasReplace got deployed
2021/12/13 11:58:03 Starting bucket ops to source bucket
2021/12/13 11:58:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:58:14 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 11:58:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:58:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:58:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:58:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:58:19 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:58:24 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:58:29 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:58:34 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 11:58:34 App: TestCasReplace got un-deployed
2021/12/13 11:58:34 No eventing-con process running
2021/12/13 11:58:34 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/13 11:58:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:58:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 11:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:58:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:58:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.19s)
=== RUN   TestCasDelete
2021/12/13 11:58:52 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:58:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 11:58:58 Waiting for app: TestCasDelete to get deployed
2021/12/13 11:59:03 Waiting for app: TestCasDelete to get deployed
2021/12/13 11:59:08 Waiting for app: TestCasDelete to get deployed
2021/12/13 11:59:13 Waiting for app: TestCasDelete to get deployed
2021/12/13 11:59:13 App: TestCasDelete got deployed
2021/12/13 11:59:13 Starting bucket ops to source bucket
2021/12/13 11:59:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:59:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 11:59:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 11:59:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 11:59:19 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 11:59:24 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 11:59:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 11:59:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 11:59:34 App: TestCasDelete got un-deployed
2021/12/13 11:59:34 No eventing-con process running
2021/12/13 11:59:34 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/13 11:59:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:59:46 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/13 11:59:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:59:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 11:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 11:59:52 Testing on the destination bucket for delete with cas
2021/12/13 11:59:52 Starting bucket ops to source bucket
2021/12/13 11:59:52 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 11:59:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 11:59:59 Waiting for app: TestCasDelete to get deployed
2021/12/13 12:00:04 Waiting for app: TestCasDelete to get deployed
2021/12/13 12:00:09 Waiting for app: TestCasDelete to get deployed
2021/12/13 12:00:14 Waiting for app: TestCasDelete to get deployed
2021/12/13 12:00:14 App: TestCasDelete got deployed
2021/12/13 12:00:14 Starting bucket ops to source bucket
2021/12/13 12:00:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:00:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:00:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:00:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 12:00:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:00:29 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 12:00:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 12:00:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 12:00:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 12:00:44 App: TestCasDelete got un-deployed
2021/12/13 12:00:44 No eventing-con process running
2021/12/13 12:00:44 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/13 12:00:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:00:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:01:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.15s)
=== RUN   TestCountersIncrement
2021/12/13 12:01:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 12:01:08 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 12:01:09 Removing node(s): [] from the cluster
2021/12/13 12:01:14 Rebalance progress: 100
2021/12/13 12:01:14 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:01:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/13 12:01:20 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:01:25 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:01:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:01:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:01:35 App: TestCountersIncrement got deployed
2021/12/13 12:01:35 Starting bucket ops to source bucket
2021/12/13 12:01:41 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 12:01:46 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/13 12:01:46 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 12:01:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:01:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:01:51 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:01:56 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:02:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:02:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:02:06 App: TestCountersIncrement got un-deployed
2021/12/13 12:02:06 No eventing-con process running
2021/12/13 12:02:06 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/13 12:02:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:02:18 Waiting for dst bucket item count to get to: 0 curr count: 678
2021/12/13 12:02:23 Waiting for dst bucket item count to get to: 0 curr count: 165
2021/12/13 12:02:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:02:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:02:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 12:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:02:35 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:02:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/13 12:02:41 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:02:46 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:02:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:02:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 12:02:56 App: TestCountersIncrement got deployed
2021/12/13 12:02:56 Starting bucket ops to source bucket
2021/12/13 12:03:02 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 12:03:02 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 12:03:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:03:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:03:07 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:03:12 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:03:17 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:03:22 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 12:03:22 App: TestCountersIncrement got un-deployed
2021/12/13 12:03:22 No eventing-con process running
2021/12/13 12:03:22 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/13 12:03:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:03:34 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/13 12:03:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 12:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:03:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:03:51 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 12:03:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 12:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:03:56 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/13 12:04:01 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.69s)
=== RUN   TestCountersDecrement
2021/12/13 12:04:01 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 12:04:01 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 12:04:02 Removing node(s): [] from the cluster
2021/12/13 12:04:07 Rebalance progress: 100
2021/12/13 12:04:07 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:04:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/13 12:04:13 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:04:18 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:04:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:04:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:04:28 App: TestCountersDecrement got deployed
2021/12/13 12:04:28 Starting bucket ops to source bucket
2021/12/13 12:04:35 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 12:04:40 Waiting for dst bucket item count to get to: 1024 curr count: 368
2021/12/13 12:04:45 Waiting for dst bucket item count to get to: 1024 curr count: 368
2021/12/13 12:04:45 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 12:04:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:04:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:04:50 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:04:55 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:05:00 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:05:05 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:05:05 App: TestCountersDecrement got un-deployed
2021/12/13 12:05:05 No eventing-con process running
2021/12/13 12:05:05 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/13 12:05:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:05:17 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/13 12:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:05:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:05:28 Waiting for dst bucket item count to get to: 0 curr count: 871
2021/12/13 12:05:33 Waiting for dst bucket item count to get to: 0 curr count: 359
2021/12/13 12:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:05:33 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:05:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/13 12:05:40 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:05:45 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:05:50 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:05:55 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 12:05:55 App: TestCountersDecrement got deployed
2021/12/13 12:05:55 Starting bucket ops to source bucket
2021/12/13 12:06:01 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 12:06:06 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/13 12:06:06 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 12:06:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:06:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:06:11 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:06:16 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:06:21 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:06:26 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 12:06:26 App: TestCountersDecrement got un-deployed
2021/12/13 12:06:26 No eventing-con process running
2021/12/13 12:06:26 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/13 12:06:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:06:38 Waiting for dst bucket item count to get to: 0 curr count: 1178
2021/12/13 12:06:43 Waiting for dst bucket item count to get to: 0 curr count: 154
2021/12/13 12:06:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:06:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:06:55 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 12:06:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:06:55 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/13 12:07:00 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.56s)
=== RUN   TestCacheGet
2021/12/13 12:07:00 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:07:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/13 12:07:06 Waiting for app: TestCacheGet to get deployed
2021/12/13 12:07:11 Waiting for app: TestCacheGet to get deployed
2021/12/13 12:07:16 Waiting for app: TestCacheGet to get deployed
2021/12/13 12:07:21 Waiting for app: TestCacheGet to get deployed
2021/12/13 12:07:21 App: TestCacheGet got deployed
2021/12/13 12:07:21 Starting bucket ops to source bucket
2021/12/13 12:07:26 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/13 12:07:31 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/13 12:07:36 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/13 12:07:41 Waiting for dst bucket item count to get to: 20 curr count: 5
2021/12/13 12:07:46 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/13 12:07:51 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/13 12:07:51 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/13 12:07:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:07:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:07:56 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 12:08:01 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 12:08:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 12:08:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 12:08:11 App: TestCacheGet got un-deployed
2021/12/13 12:08:11 No eventing-con process running
2021/12/13 12:08:11 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/13 12:08:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:08:23 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/13 12:08:28 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/13 12:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:08:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:08:40 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/13 12:08:45 Waiting for dst bucket item count to get to: 0 curr count: 11
2021/12/13 12:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (105.20s)
=== RUN   TestCacheRYOW
2021/12/13 12:08:45 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:08:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/13 12:08:51 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 12:08:56 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 12:09:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 12:09:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 12:09:06 App: TestCacheRYOW got deployed
2021/12/13 12:09:06 Starting bucket ops to source bucket
2021/12/13 12:09:11 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/13 12:09:11 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/13 12:09:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:09:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:09:16 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 12:09:21 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 12:09:26 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 12:09:31 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 12:09:31 App: TestCacheRYOW got un-deployed
2021/12/13 12:09:31 No eventing-con process running
2021/12/13 12:09:31 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/13 12:09:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:09:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:09:48 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 12:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:10:00 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/13 12:10:05 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/13 12:10:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.24s)
=== RUN   TestCacheExpiry
2021/12/13 12:10:05 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:10:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/13 12:10:11 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 12:10:16 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 12:10:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 12:10:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 12:10:26 App: TestCacheExpiry got deployed
2021/12/13 12:10:26 Starting bucket ops to source bucket
2021/12/13 12:10:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:10:36 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 12:10:41 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 12:10:46 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 12:10:51 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 12:10:51 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:10:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:10:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:10:57 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 12:11:02 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 12:11:07 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 12:11:12 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 12:11:12 App: TestCacheExpiry got un-deployed
2021/12/13 12:11:12 No eventing-con process running
2021/12/13 12:11:12 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/13 12:11:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:11:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:11:28 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 12:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:11:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:11:40 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/13 12:11:45 Waiting for dst bucket item count to get to: 0 curr count: 73
2021/12/13 12:11:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.18s)
=== RUN   TestCacheExpiryCustom
2021/12/13 12:11:45 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:11:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/13 12:11:51 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 12:11:56 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 12:12:01 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 12:12:07 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 12:12:07 App: TestCacheExpiryCustom got deployed
2021/12/13 12:12:07 Starting bucket ops to source bucket
2021/12/13 12:12:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:12:17 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 12:12:22 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 12:12:27 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 12:12:32 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 12:12:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:12:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:12:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:12:37 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 12:12:42 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 12:12:47 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 12:12:52 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 12:12:52 App: TestCacheExpiryCustom got un-deployed
2021/12/13 12:12:52 No eventing-con process running
2021/12/13 12:12:52 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/13 12:12:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:13:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:13:09 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 12:13:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:13:20 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/13 12:13:25 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/13 12:13:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.26s)
=== RUN   TestCacheOverflowCustom
2021/12/13 12:13:25 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:13:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/13 12:13:32 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 12:13:37 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 12:13:42 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 12:13:47 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 12:13:47 App: TestCacheOverflowCustom got deployed
2021/12/13 12:13:47 Starting bucket ops to source bucket
2021/12/13 12:13:52 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/13 12:13:57 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/13 12:14:02 Waiting for dst bucket item count to get to: 35 curr count: 6
2021/12/13 12:14:07 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/13 12:14:12 Waiting for dst bucket item count to get to: 35 curr count: 26
2021/12/13 12:14:12 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/13 12:14:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:14:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:14:17 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 12:14:22 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 12:14:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 12:14:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 12:14:32 App: TestCacheOverflowCustom got un-deployed
2021/12/13 12:14:32 No eventing-con process running
2021/12/13 12:14:32 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/13 12:14:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:14:44 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/13 12:14:49 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/13 12:14:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:14:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:15:01 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/13 12:15:06 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/13 12:15:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.23s)
=== RUN   TestStrictMode
2021/12/13 12:15:06 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:15:07 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/13 12:15:07 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:15:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/13 12:15:13 Waiting for app: TestDataTypes to get deployed
2021/12/13 12:15:18 Waiting for app: TestDataTypes to get deployed
2021/12/13 12:15:23 Waiting for app: TestDataTypes to get deployed
2021/12/13 12:15:28 Waiting for app: TestDataTypes to get deployed
2021/12/13 12:15:28 App: TestDataTypes got deployed
2021/12/13 12:15:28 Starting bucket ops to source bucket
2021/12/13 12:15:33 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/13 12:15:38 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/13 12:15:43 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/13 12:15:43 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/13 12:15:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:15:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:15:48 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 12:15:53 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 12:15:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 12:16:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 12:16:03 App: TestDataTypes got un-deployed
2021/12/13 12:16:03 No eventing-con process running
2021/12/13 12:16:03 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/13 12:16:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:16:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 12:16:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:16:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:16:27 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/13 12:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.06s)
=== RUN   TestEnoentGet
2021/12/13 12:16:27 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:16:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/13 12:16:33 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:16:38 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:16:43 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:16:48 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:16:48 App: TestEnoentGet got deployed
2021/12/13 12:16:48 Starting bucket ops to source bucket
2021/12/13 12:16:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:16:58 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 12:16:58 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:16:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:16:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:17:03 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:17:08 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:17:13 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:17:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:17:18 App: TestEnoentGet got un-deployed
2021/12/13 12:17:18 No eventing-con process running
2021/12/13 12:17:18 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/13 12:17:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:17:30 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/13 12:17:35 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/13 12:17:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:17:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:17:47 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:17:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/13 12:17:53 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:17:58 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:18:03 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:18:08 Waiting for app: TestEnoentGet to get deployed
2021/12/13 12:18:08 App: TestEnoentGet got deployed
2021/12/13 12:18:08 Starting bucket ops to source bucket
2021/12/13 12:18:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:18:18 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 12:18:18 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:18:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:18:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:18:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:18:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:18:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:18:38 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 12:18:38 App: TestEnoentGet got un-deployed
2021/12/13 12:18:38 No eventing-con process running
2021/12/13 12:18:39 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/13 12:18:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:18:50 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/13 12:18:55 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/13 12:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:18:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:19:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:19:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.26s)
=== RUN   TestEnoentDelete
2021/12/13 12:19:07 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:19:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 12:19:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:19:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:19:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:19:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:19:28 App: TestEnoentDelete got deployed
2021/12/13 12:19:28 Starting bucket ops to source bucket
2021/12/13 12:19:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:19:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 12:19:38 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:19:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:19:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:19:44 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:19:49 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:19:54 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:19:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:19:59 App: TestEnoentDelete got un-deployed
2021/12/13 12:19:59 No eventing-con process running
2021/12/13 12:19:59 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/13 12:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:20:10 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/13 12:20:15 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/13 12:20:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:20:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:20:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:20:27 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:20:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 12:20:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:20:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:20:43 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:20:48 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 12:20:48 App: TestEnoentDelete got deployed
2021/12/13 12:20:48 Starting bucket ops to source bucket
2021/12/13 12:20:54 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:20:59 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 12:20:59 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:20:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:20:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:21:04 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:21:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:21:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:21:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 12:21:19 App: TestEnoentDelete got un-deployed
2021/12/13 12:21:19 No eventing-con process running
2021/12/13 12:21:19 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/13 12:21:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:21:31 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/13 12:21:36 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/13 12:21:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:21:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:21:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.32s)
=== RUN   TestError
2021/12/13 12:21:57 Waiting for app: TestError to get un-deployed
2021/12/13 12:21:57 App: TestError got un-deployed
2021/12/13 12:21:57 No eventing-con process running
2021/12/13 12:21:57 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/13 12:21:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:22:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:22:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:22:11 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:22:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/13 12:22:17 Waiting for app: TestError to get deployed
2021/12/13 12:22:22 Waiting for app: TestError to get deployed
2021/12/13 12:22:27 Waiting for app: TestError to get deployed
2021/12/13 12:22:32 Waiting for app: TestError to get deployed
2021/12/13 12:22:32 App: TestError got deployed
2021/12/13 12:22:32 Starting bucket ops to source bucket
2021/12/13 12:22:37 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/13 12:22:42 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/13 12:22:42 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/13 12:22:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:22:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:22:48 Waiting for app: TestError to get un-deployed
2021/12/13 12:22:53 Waiting for app: TestError to get un-deployed
2021/12/13 12:22:58 Waiting for app: TestError to get un-deployed
2021/12/13 12:23:03 Waiting for app: TestError to get un-deployed
2021/12/13 12:23:03 App: TestError got un-deployed
2021/12/13 12:23:03 No eventing-con process running
2021/12/13 12:23:03 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/13 12:23:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:23:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:23:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:23:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:23:26 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/13 12:23:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (98.79s)
=== RUN   TestCRLF
2021/12/13 12:23:36 Waiting for app: TestCRLF to get un-deployed
2021/12/13 12:23:36 App: TestCRLF got un-deployed
2021/12/13 12:23:36 No eventing-con process running
2021/12/13 12:23:36 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/13 12:23:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:23:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:23:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:23:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:23:50 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:23:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/13 12:23:56 Waiting for app: TestCRLF to get deployed
2021/12/13 12:24:01 Waiting for app: TestCRLF to get deployed
2021/12/13 12:24:06 Waiting for app: TestCRLF to get deployed
2021/12/13 12:24:11 Waiting for app: TestCRLF to get deployed
2021/12/13 12:24:11 App: TestCRLF got deployed
2021/12/13 12:24:11 Starting bucket ops to source bucket
2021/12/13 12:24:16 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:24:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 12:24:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 12:24:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:24:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:24:26 Waiting for app: TestCRLF to get un-deployed
2021/12/13 12:24:31 Waiting for app: TestCRLF to get un-deployed
2021/12/13 12:24:36 Waiting for app: TestCRLF to get un-deployed
2021/12/13 12:24:41 Waiting for app: TestCRLF to get un-deployed
2021/12/13 12:24:41 App: TestCRLF got un-deployed
2021/12/13 12:24:41 No eventing-con process running
2021/12/13 12:24:41 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/13 12:24:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:24:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 12:24:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 12:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:25:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:25:10 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/13 12:25:15 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/13 12:25:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.72s)
=== RUN   TestImportExport
2021/12/13 12:25:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:25:25 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 12:25:25 No eventing-con process running
2021/12/13 12:25:25 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/13 12:25:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:25:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:25:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:25:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:25:44 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:25:44 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 12:25:44 No eventing-con process running
2021/12/13 12:25:44 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/13 12:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:25:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:25:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:25:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:25:57 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:25:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 12:25:58 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:25:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 12:26:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 12:26:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 12:26:15 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 12:26:20 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 12:26:20 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/13 12:26:25 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 12:26:30 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 12:26:35 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 12:26:35 App: TestImportExport_bucket_op_on_update got deployed
2021/12/13 12:26:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:26:40 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:26:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:26:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:26:55 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:26:55 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 12:26:55 No eventing-con process running
2021/12/13 12:26:55 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/13 12:26:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:27:00 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:27:05 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:27:10 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:27:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:27:15 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 12:27:15 No eventing-con process running
2021/12/13 12:27:15 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/13 12:27:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 12:27:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 12:27:32 No eventing-con process running
2021/12/13 12:27:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/13 12:27:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:27:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:27:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 12:27:51 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 12:27:51 No eventing-con process running
2021/12/13 12:27:51 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/13 12:27:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:27:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:27:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:28:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.59s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/13 12:28:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:28:15 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:28:15 No eventing-con process running
2021/12/13 12:28:15 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/13 12:28:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:28:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:28:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:28:28 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:28:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:28:29 Starting bucket ops to source bucket
2021/12/13 12:28:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:28:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:28:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:28:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:28:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 12:28:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:29:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 12:29:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 12:29:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:29:02 Undeploying app: bucket_op_on_update
2021/12/13 12:29:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:29:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:29:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:29:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:29:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:29:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:29:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:29:22 No eventing-con process running
2021/12/13 12:29:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:29:31 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:30:07 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:30:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:30:08 Starting bucket ops to source bucket
2021/12/13 12:30:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:30:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:30:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:30:31 Waiting for dst bucket item count to get to: 5000 curr count: 283
2021/12/13 12:30:36 Waiting for dst bucket item count to get to: 5000 curr count: 283
2021/12/13 12:30:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:30:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 12:30:41 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 12:30:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:30:41 Undeploying app: bucket_op_on_update
2021/12/13 12:30:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:30:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:30:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:30:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:31:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:31:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:31:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:31:01 No eventing-con process running
2021/12/13 12:31:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:31:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:31:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:31:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:31:46 Starting bucket ops to source bucket
2021/12/13 12:31:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:31:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:32:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:32:09 Waiting for dst bucket item count to get to: 5000 curr count: 2227
2021/12/13 12:32:14 Waiting for dst bucket item count to get to: 5000 curr count: 2227
2021/12/13 12:32:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:32:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 12:32:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 12:32:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:32:19 Undeploying app: bucket_op_on_update
2021/12/13 12:32:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:32:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:32:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:32:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:32:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:32:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:32:39 No eventing-con process running
2021/12/13 12:32:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:32:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:33:23 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:33:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:33:24 Starting bucket ops to source bucket
2021/12/13 12:33:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:33:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:33:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:33:47 Waiting for dst bucket item count to get to: 5000 curr count: 1039
2021/12/13 12:33:52 Waiting for dst bucket item count to get to: 5000 curr count: 3539
2021/12/13 12:33:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:33:57 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 12:33:57 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 12:33:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:33:57 Undeploying app: bucket_op_on_update
2021/12/13 12:33:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:34:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:34:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:34:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:34:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:34:18 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:34:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:34:18 No eventing-con process running
2021/12/13 12:34:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:34:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:35:02 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:35:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:35:03 Starting bucket ops to source bucket
2021/12/13 12:35:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:35:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:35:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:35:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:35:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:35:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 12:35:31 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 12:35:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:35:31 Undeploying app: bucket_op_on_update
2021/12/13 12:35:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:35:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:35:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:35:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:35:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 12:35:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 12:35:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:35:51 No eventing-con process running
2021/12/13 12:35:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:36:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:36:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.53s)
=== RUN   TestOnUpdateN1QLOp
2021/12/13 12:36:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 12:36:45 App: TestOnUpdateN1QLOp got un-deployed
2021/12/13 12:36:45 No eventing-con process running
2021/12/13 12:36:45 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/13 12:36:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:36:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:36:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:36:59 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:37:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/13 12:37:00 Starting bucket ops to source bucket
2021/12/13 12:37:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:37:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:37:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:37:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:37:28 Waiting for dst bucket item count to get to: 5000 curr count: 2459
2021/12/13 12:37:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:37:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:37:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:37:33 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 12:37:38 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 12:37:43 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 12:37:48 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 12:37:48 App: TestOnUpdateN1QLOp got un-deployed
2021/12/13 12:37:48 No eventing-con process running
2021/12/13 12:37:48 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/13 12:37:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:38:00 Waiting for dst bucket item count to get to: 0 curr count: 2919
2021/12/13 12:38:05 Waiting for dst bucket item count to get to: 0 curr count: 2919
2021/12/13 12:38:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:38:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:38:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.79s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/13 12:38:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 12:38:27 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/13 12:38:27 No eventing-con process running
2021/12/13 12:38:27 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/13 12:38:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:38:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:38:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:38:40 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:38:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:38:42 Starting bucket ops to source bucket
2021/12/13 12:38:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:38:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:38:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:39:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:39:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:39:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:39:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:39:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 12:39:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 12:39:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 12:39:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 12:39:25 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/13 12:39:25 No eventing-con process running
2021/12/13 12:39:25 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/13 12:39:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:39:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:39:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:39:48 Waiting for dst bucket item count to get to: 0 curr count: 4065
2021/12/13 12:39:53 Waiting for dst bucket item count to get to: 0 curr count: 1565
2021/12/13 12:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.51s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/13 12:40:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 12:40:03 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 12:40:03 No eventing-con process running
2021/12/13 12:40:03 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/13 12:40:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:40:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:40:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:40:17 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:40:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 12:40:18 Starting bucket ops to source bucket
2021/12/13 12:40:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:40:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:40:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:40:41 Waiting for dst bucket item count to get to: 5000 curr count: 257
2021/12/13 12:40:46 Waiting for dst bucket item count to get to: 5000 curr count: 257
2021/12/13 12:40:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:40:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:40:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:40:51 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 12:40:56 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 12:41:01 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 12:41:06 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 12:41:06 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 12:41:06 No eventing-con process running
2021/12/13 12:41:06 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/13 12:41:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:41:18 Waiting for dst bucket item count to get to: 0 curr count: 3453
2021/12/13 12:41:23 Waiting for dst bucket item count to get to: 0 curr count: 953
2021/12/13 12:41:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:41:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:41:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.58s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/13 12:41:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 12:41:45 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/13 12:41:45 No eventing-con process running
2021/12/13 12:41:45 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/13 12:41:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:41:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:41:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:41:58 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:42:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/13 12:42:00 Starting bucket ops to source bucket
2021/12/13 12:42:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:42:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:42:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:42:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:42:28 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 12:42:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:42:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:42:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:42:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 12:42:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 12:42:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 12:42:48 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 12:42:48 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/13 12:42:48 No eventing-con process running
2021/12/13 12:42:48 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/13 12:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:43:00 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 12:43:05 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 12:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:43:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:43:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:43:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.68s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/13 12:43:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 12:43:27 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/13 12:43:27 No eventing-con process running
2021/12/13 12:43:27 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/13 12:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:43:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:43:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:43:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:43:40 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:43:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/13 12:43:42 Starting bucket ops to source bucket
2021/12/13 12:43:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:43:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:44:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:44:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:44:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:44:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:44:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:44:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 12:44:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 12:44:20 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 12:44:25 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 12:44:25 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/13 12:44:25 No eventing-con process running
2021/12/13 12:44:25 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/13 12:44:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:44:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:44:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:44:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:44:49 Waiting for dst bucket item count to get to: 0 curr count: 4798
2021/12/13 12:44:54 Waiting for dst bucket item count to get to: 0 curr count: 2298
2021/12/13 12:44:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.20s)
=== RUN   TestOnDeleteBucketOp
2021/12/13 12:45:04 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 12:45:04 App: TestOnDeleteBucketOp got un-deployed
2021/12/13 12:45:04 No eventing-con process running
2021/12/13 12:45:04 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/13 12:45:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:45:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:45:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:45:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:45:17 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:45:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 12:45:19 Starting bucket ops to source bucket
2021/12/13 12:45:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:45:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:45:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:45:44 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/13 12:45:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:45:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:45:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:45:50 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 12:45:55 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 12:46:00 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 12:46:05 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 12:46:05 App: TestOnDeleteBucketOp got un-deployed
2021/12/13 12:46:05 No eventing-con process running
2021/12/13 12:46:05 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/13 12:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:46:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:46:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:46:24 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:46:29 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/13 12:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.97s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/13 12:46:39 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 12:46:39 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/13 12:46:39 No eventing-con process running
2021/12/13 12:46:39 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/13 12:46:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:46:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:46:52 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:46:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/13 12:46:54 Starting bucket ops to source bucket
2021/12/13 12:47:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:47:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:47:14 Waiting for dst bucket item count to get to: 5000 curr count: 2351
2021/12/13 12:47:19 Waiting for dst bucket item count to get to: 5000 curr count: 4847
2021/12/13 12:47:19 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:47:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:47:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:47:25 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 12:47:30 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 12:47:35 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 12:47:40 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 12:47:40 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/13 12:47:40 No eventing-con process running
2021/12/13 12:47:40 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/13 12:47:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:47:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:47:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:47:59 Waiting for dst bucket item count to get to: 0 curr count: 4130
2021/12/13 12:48:04 Waiting for dst bucket item count to get to: 0 curr count: 1634
2021/12/13 12:48:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (95.00s)
=== RUN   TestTimerBucketOp
2021/12/13 12:48:14 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 12:48:14 App: TestTimerBucketOp got un-deployed
2021/12/13 12:48:14 No eventing-con process running
2021/12/13 12:48:14 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/13 12:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:48:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:48:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:48:27 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:48:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 12:48:28 Starting bucket ops to source bucket
2021/12/13 12:48:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:48:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:48:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:48:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:48:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:49:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:49:06 Waiting for dst bucket item count to get to: 5000 curr count: 2358
2021/12/13 12:49:11 Waiting for dst bucket item count to get to: 5000 curr count: 4858
2021/12/13 12:49:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:49:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:49:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:49:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 12:49:22 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 12:49:27 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 12:49:32 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 12:49:32 App: TestTimerBucketOp got un-deployed
2021/12/13 12:49:32 No eventing-con process running
2021/12/13 12:49:32 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/13 12:49:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:49:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:49:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 12:49:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:49:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:50:00 Waiting for dst bucket item count to get to: 0 curr count: 3491
2021/12/13 12:50:05 Waiting for dst bucket item count to get to: 0 curr count: 3491
2021/12/13 12:50:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.40s)
=== RUN   TestTimerInPastBucketOp
2021/12/13 12:50:15 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 12:50:15 App: TestTimerInPastBucketOp got un-deployed
2021/12/13 12:50:15 No eventing-con process running
2021/12/13 12:50:15 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/13 12:50:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:50:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:50:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:50:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:50:29 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:50:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 12:50:30 Starting bucket ops to source bucket
2021/12/13 12:50:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:50:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:50:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:50:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:50:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:51:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:51:08 Waiting for dst bucket item count to get to: 5000 curr count: 915
2021/12/13 12:51:13 Waiting for dst bucket item count to get to: 5000 curr count: 3055
2021/12/13 12:51:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:51:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:51:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:51:18 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 12:51:23 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 12:51:28 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 12:51:33 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 12:51:33 App: TestTimerInPastBucketOp got un-deployed
2021/12/13 12:51:33 No eventing-con process running
2021/12/13 12:51:33 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/13 12:51:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:51:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:51:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:51:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:51:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 12:51:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.50s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/13 12:52:07 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:52:07 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 12:52:07 No eventing-con process running
2021/12/13 12:52:07 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/13 12:52:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:52:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:52:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:52:20 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:52:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 12:52:21 Starting bucket ops to source bucket
2021/12/13 12:52:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:52:59 Waiting for dst bucket item count to get to: 5000 curr count: 1715
2021/12/13 12:53:04 Waiting for dst bucket item count to get to: 5000 curr count: 1715
2021/12/13 12:53:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:53:09 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 12:53:09 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 12:53:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:53:09 Undeploying app: bucket_op_with_timer
2021/12/13 12:53:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:53:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:53:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:53:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:53:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:53:29 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 12:53:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:53:30 No eventing-con process running
2021/12/13 12:53:31 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:53:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:54:13 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:54:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 12:54:15 Starting bucket ops to source bucket
2021/12/13 12:54:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:54:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:54:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:54:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:54:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:54:48 Waiting for dst bucket item count to get to: 5000 curr count: 434
2021/12/13 12:54:53 Waiting for dst bucket item count to get to: 5000 curr count: 1494
2021/12/13 12:54:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:54:58 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 12:54:58 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 12:54:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:54:58 Undeploying app: bucket_op_with_timer
2021/12/13 12:54:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:55:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:55:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:55:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:55:18 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:55:18 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 12:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:55:18 No eventing-con process running
2021/12/13 12:55:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:55:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:56:02 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:56:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 12:56:03 Starting bucket ops to source bucket
2021/12/13 12:56:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:56:46 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 12:56:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:56:51 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 12:56:51 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 12:56:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:56:51 Undeploying app: bucket_op_with_timer
2021/12/13 12:56:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:56:56 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:57:01 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:57:06 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:57:11 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:57:11 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 12:57:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:57:11 No eventing-con process running
2021/12/13 12:57:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:57:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:57:55 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:57:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 12:57:56 Starting bucket ops to source bucket
2021/12/13 12:58:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:58:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 12:58:39 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 12:58:39 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 12:58:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 12:58:39 Undeploying app: bucket_op_with_timer
2021/12/13 12:58:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 12:58:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:58:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:58:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:59:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 12:59:00 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 12:59:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 12:59:00 No eventing-con process running
2021/12/13 12:59:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:59:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 12:59:44 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 12:59:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 12:59:45 Starting bucket ops to source bucket
2021/12/13 12:59:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 12:59:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:00:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:00:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:00:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:00:18 Waiting for dst bucket item count to get to: 5000 curr count: 260
2021/12/13 13:00:23 Waiting for dst bucket item count to get to: 5000 curr count: 260
2021/12/13 13:00:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:00:28 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 13:00:28 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 13:00:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:00:28 Undeploying app: bucket_op_with_timer
2021/12/13 13:00:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:00:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 13:00:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 13:00:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 13:00:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 13:00:48 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 13:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:00:48 No eventing-con process running
2021/12/13 13:00:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:01:32 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (575.43s)
=== RUN   TestMultipleHandlers
2021/12/13 13:01:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 13:01:42 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/13 13:01:42 No eventing-con process running
2021/12/13 13:01:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/13 13:01:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:01:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:02:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 13:02:01 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/13 13:02:01 No eventing-con process running
2021/12/13 13:02:01 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/13 13:02:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:02:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:02:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:02:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:02:14 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:02:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 13:02:15 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:02:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/13 13:02:22 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 13:02:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 13:02:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 13:02:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 13:02:37 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/13 13:02:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 13:02:47 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 13:02:52 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 13:02:52 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/13 13:02:52 Starting bucket ops to source bucket
2021/12/13 13:03:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:03:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:03:02 Starting bucket ops to source bucket
2021/12/13 13:03:16 Waiting for dst bucket item count to get to: 10000 curr count: 6840
2021/12/13 13:03:21 Waiting for dst bucket item count to get to: 10000 curr count: 6840
2021/12/13 13:03:21 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:03:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:03:21 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:03:22 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:03:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:03:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 13:03:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 13:03:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 13:03:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 13:03:42 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/13 13:03:42 No eventing-con process running
2021/12/13 13:03:42 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/13 13:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:03:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:04:01 Waiting for dst bucket item count to get to: 0 curr count: 8184
2021/12/13 13:04:06 Waiting for dst bucket item count to get to: 0 curr count: 8184
2021/12/13 13:04:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:04:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:04:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 13:04:16 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 13:04:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 13:04:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 13:04:26 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/13 13:04:26 No eventing-con process running
2021/12/13 13:04:26 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/13 13:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:04:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:04:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:04:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.42s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/13 13:04:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 13:04:50 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/13 13:04:50 No eventing-con process running
2021/12/13 13:04:50 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/13 13:04:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:04:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:04:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:05:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:05:03 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:05:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 13:05:04 Starting bucket ops to source bucket
2021/12/13 13:05:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:05:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:05:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:05:27 Waiting for dst bucket item count to get to: 5000 curr count: 1637
2021/12/13 13:05:32 Waiting for dst bucket item count to get to: 5000 curr count: 4137
2021/12/13 13:05:32 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:05:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:05:37 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 13:05:37 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 13:05:37 Pausing app: bucket_op_on_update
2021/12/13 13:05:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:05:42 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 13:05:42 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 13:05:42 Resuming app: bucket_op_on_update
2021/12/13 13:05:43 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:05:43 Starting bucket ops to source bucket
2021/12/13 13:05:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:05:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:06:01 Waiting for dst bucket item count to get to: 10000 curr count: 5043
2021/12/13 13:06:06 Waiting for dst bucket item count to get to: 10000 curr count: 7543
2021/12/13 13:06:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:06:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:06:12 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 13:06:12 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 13:06:12 Pausing app: bucket_op_on_update
2021/12/13 13:06:12 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:06:17 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 13:06:17 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 13:06:17 Resuming app: bucket_op_on_update
2021/12/13 13:06:18 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:06:18 Starting bucket ops to source bucket
2021/12/13 13:06:26 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:06:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:06:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/13 13:06:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:06:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 13:06:36 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 13:06:36 Pausing app: bucket_op_on_update
2021/12/13 13:06:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:06:41 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 13:06:41 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 13:06:41 Resuming app: bucket_op_on_update
2021/12/13 13:06:42 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:06:42 Starting bucket ops to source bucket
2021/12/13 13:06:50 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:06:55 Waiting for dst bucket item count to get to: 20000 curr count: 15721
2021/12/13 13:07:00 Waiting for dst bucket item count to get to: 20000 curr count: 15721
2021/12/13 13:07:05 Waiting for dst bucket item count to get to: 20000 curr count: 15721
2021/12/13 13:07:05 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 13:07:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:07:10 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 13:07:10 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 13:07:10 Pausing app: bucket_op_on_update
2021/12/13 13:07:10 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:07:15 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 13:07:15 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 13:07:15 Resuming app: bucket_op_on_update
2021/12/13 13:07:16 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:07:16 Starting bucket ops to source bucket
2021/12/13 13:07:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:07:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:07:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:07:34 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 13:07:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:07:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 13:07:39 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 13:07:39 Pausing app: bucket_op_on_update
2021/12/13 13:07:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:07:45 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 13:07:45 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 13:07:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:07:50 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 13:07:55 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 13:08:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 13:08:05 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 13:08:05 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/13 13:08:05 No eventing-con process running
2021/12/13 13:08:05 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/13 13:08:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:08:17 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 13:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:08:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:08:28 Waiting for dst bucket item count to get to: 0 curr count: 20929
2021/12/13 13:08:33 Waiting for dst bucket item count to get to: 0 curr count: 8429
2021/12/13 13:08:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.84s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/13 13:08:43 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 13:08:43 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/13 13:08:43 No eventing-con process running
2021/12/13 13:08:43 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/13 13:08:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:08:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:08:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:08:57 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:08:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 13:08:58 Starting bucket ops to source bucket
2021/12/13 13:09:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:09:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:09:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:09:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:09:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:09:31 Waiting for dst bucket item count to get to: 5000 curr count: 7
2021/12/13 13:09:36 Waiting for dst bucket item count to get to: 5000 curr count: 2507
2021/12/13 13:09:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:09:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:09:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 13:09:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 13:09:41 Pausing app: bucket_op_with_timer
2021/12/13 13:09:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:09:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 13:09:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 13:09:46 Resuming app: bucket_op_with_timer
2021/12/13 13:09:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:09:47 Starting bucket ops to source bucket
2021/12/13 13:09:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:10:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:10:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:10:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:10:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:10:21 Waiting for dst bucket item count to get to: 10000 curr count: 5802
2021/12/13 13:10:26 Waiting for dst bucket item count to get to: 10000 curr count: 8302
2021/12/13 13:10:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:10:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:10:31 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 13:10:31 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 13:10:31 Pausing app: bucket_op_with_timer
2021/12/13 13:10:31 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:10:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 13:10:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 13:10:36 Resuming app: bucket_op_with_timer
2021/12/13 13:10:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:10:37 Starting bucket ops to source bucket
2021/12/13 13:10:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:10:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:10:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:11:00 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:11:05 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 13:11:10 Waiting for dst bucket item count to get to: 15000 curr count: 11690
2021/12/13 13:11:15 Waiting for dst bucket item count to get to: 15000 curr count: 14190
2021/12/13 13:11:15 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/13 13:11:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:11:20 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 13:11:20 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 13:11:20 Pausing app: bucket_op_with_timer
2021/12/13 13:11:20 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:11:25 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 13:11:25 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 13:11:25 Resuming app: bucket_op_with_timer
2021/12/13 13:11:26 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:11:26 Starting bucket ops to source bucket
2021/12/13 13:11:34 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:11:39 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:11:44 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:11:49 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:11:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 13:11:59 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/13 13:12:04 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/13 13:12:04 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 13:12:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:12:09 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 13:12:09 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 13:12:09 Pausing app: bucket_op_with_timer
2021/12/13 13:12:09 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:12:14 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 13:12:14 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 13:12:14 Resuming app: bucket_op_with_timer
2021/12/13 13:12:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:12:16 Starting bucket ops to source bucket
2021/12/13 13:12:24 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:12:29 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:12:34 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:12:39 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:12:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 13:12:49 Waiting for dst bucket item count to get to: 25000 curr count: 23114
2021/12/13 13:12:54 Waiting for dst bucket item count to get to: 25000 curr count: 23114
2021/12/13 13:12:54 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 13:12:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:12:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 13:12:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 13:12:59 Pausing app: bucket_op_with_timer
2021/12/13 13:12:59 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:13:04 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 13:13:04 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 13:13:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:13:09 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 13:13:14 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 13:13:19 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 13:13:24 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 13:13:24 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/13 13:13:24 No eventing-con process running
2021/12/13 13:13:24 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/13 13:13:26 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:13:36 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 13:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:13:48 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/13 13:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (314.21s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/13 13:13:58 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 13:13:58 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/13 13:13:58 No eventing-con process running
2021/12/13 13:13:58 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/13 13:13:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:14:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:14:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:14:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:14:11 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:14:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/13 13:14:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 13:14:22 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 13:14:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 13:14:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 13:14:32 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/13 13:14:32 Starting bucket ops to source bucket
2021/12/13 13:14:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:14:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:14:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:14:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:15:02 Waiting for dst bucket item count to get to: 5000 curr count: 4022
2021/12/13 13:15:07 Waiting for dst bucket item count to get to: 5000 curr count: 4022
2021/12/13 13:15:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:15:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:15:07 Pausing app: bucket_op_with_timer
2021/12/13 13:15:07 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:15:12 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/13 13:15:12 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/13 13:15:12 Resuming app: bucket_op_with_timer
2021/12/13 13:15:13 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:15:13 Starting bucket ops to source bucket
2021/12/13 13:15:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:15:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:15:35 Waiting for dst bucket item count to get to: 10000 curr count: 5008
2021/12/13 13:15:40 Waiting for dst bucket item count to get to: 10000 curr count: 5127
2021/12/13 13:15:45 Waiting for dst bucket item count to get to: 10000 curr count: 7029
2021/12/13 13:15:50 Waiting for dst bucket item count to get to: 10000 curr count: 9410
2021/12/13 13:15:50 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:15:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:15:55 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 13:16:00 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 13:16:05 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 13:16:10 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 13:16:10 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/13 13:16:10 No eventing-con process running
2021/12/13 13:16:10 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/13 13:16:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:16:22 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 13:16:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 13:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:16:38 Waiting for dst bucket item count to get to: 0 curr count: 8543
2021/12/13 13:16:43 Waiting for dst bucket item count to get to: 0 curr count: 3543
2021/12/13 13:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (175.87s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/13 13:16:53 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 13:16:53 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/13 13:16:53 No eventing-con process running
2021/12/13 13:16:53 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/13 13:16:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:17:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:17:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:17:07 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:17:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 13:17:13 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 13:17:18 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 13:17:23 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 13:17:28 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 13:17:28 App: TestPauseResumeWithEventingReb got deployed
2021/12/13 13:17:28 Starting bucket ops to source bucket
2021/12/13 13:17:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:17:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:17:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:17:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:17:58 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 13:17:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:17:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:17:58 Pausing app: bucket_op_with_timer
2021/12/13 13:17:58 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:18:03 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/13 13:18:03 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/13 13:18:03 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 13:18:03 Removing node(s): [] from the cluster
2021/12/13 13:18:08 Rebalance progress: 29.16666666666666
2021/12/13 13:18:13 Rebalance progress: 29.16666666666666
2021/12/13 13:18:18 Rebalance progress: 100
2021/12/13 13:18:18 Resuming app: bucket_op_with_timer
2021/12/13 13:18:19 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:18:19 Starting bucket ops to source bucket
2021/12/13 13:18:31 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:18:36 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:18:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:18:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:18:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 13:18:56 Waiting for dst bucket item count to get to: 10000 curr count: 6897
2021/12/13 13:19:01 Waiting for dst bucket item count to get to: 10000 curr count: 9397
2021/12/13 13:19:01 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:19:01 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 13:19:06 Rebalance progress: 29.16666666666666
2021/12/13 13:19:11 Rebalance progress: 29.16666666666666
2021/12/13 13:19:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/13 13:19:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:19:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:19:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:20:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:20:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:20:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:20:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:21:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:21:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:21:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 13:21:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:63} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 765e173bddfa85df96c0f24f323a8118
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: 4daac716440afe4f61f86e37322a59f4
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
Producer node: 127.0.0.1:9303	Node UUID: b5363cb4026a70b75c32112801f4038c
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/13 13:22:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:22:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:22:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 13:22:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 13:22:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 13:22:27 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 13:22:27 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/13 13:22:27 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:22:27 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/13 13:22:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:22:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9411
2021/12/13 13:22:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4411
2021/12/13 13:22:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:22:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:22:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 13:22:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (372.15s)
=== RUN   TestBucketDeleteAfterPause
2021/12/13 13:23:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/13 13:23:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/13 13:23:06 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:23:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/13 13:23:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:23:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:23:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:23:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:23:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:23:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/13 13:23:25 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 13:23:30 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 13:23:35 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 13:23:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 13:23:40 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/13 13:23:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:23:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:23:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:23:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:23:50 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/13 13:23:50 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:23:55 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/13 13:23:55 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/13 13:23:55 handler_test.go:716: Deleting source bucket: default
2021/12/13 13:24:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/13 13:24:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/13 13:24:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/13 13:24:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/13 13:24:07 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:24:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/13 13:24:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:24:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:24:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:24:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:24:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3238
2021/12/13 13:24:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3238
2021/12/13 13:24:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.47s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/13 13:24:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 13:24:45 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/13 13:24:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:24:45 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/13 13:24:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:24:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:24:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:24:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:24:58 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:24:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/13 13:25:04 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 13:25:09 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 13:25:14 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 13:25:19 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 13:25:19 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/13 13:25:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:25:54 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/13 13:25:54 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:25:59 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/13 13:25:59 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/13 13:28:59 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/13 13:28:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/13 13:29:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/13 13:29:05 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 13:29:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:29:05 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/13 13:29:10 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 13:29:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:29:11 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2958 expected 5000
2021/12/13 13:29:16 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 13:29:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:29:16 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/13 13:29:21 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 13:29:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:29:21 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/13 13:29:21 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/13 13:29:21 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:29:31 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 13:29:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 13:29:41 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 13:29:41 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/13 13:29:41 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:29:41 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/13 13:29:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:29:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:29:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:29:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:29:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:30:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.78s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/13 13:30:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 13:30:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/13 13:30:14 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:30:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/13 13:30:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:30:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:30:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:30:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:30:27 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:30:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 13:30:33 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:30:38 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:30:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:30:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:30:48 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/13 13:30:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:30:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:31:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 611
2021/12/13 13:31:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3111
2021/12/13 13:31:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:31:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:31:07 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/13 13:31:07 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:31:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/13 13:31:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 13:31:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:31:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:31:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:31:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 13:31:58 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/13 13:31:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:32:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:32:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:32:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:32:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:32:16 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/13 13:32:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:32:26 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 13:32:31 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 13:32:36 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 13:32:36 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/13 13:32:36 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:32:37 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/13 13:32:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:32:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:32:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:32:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.73s)
=== RUN   TestCPPWorkerCleanup
2021/12/13 13:33:00 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 13:33:00 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/13 13:33:00 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:33:00 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/13 13:33:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:33:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:33:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:33:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:33:13 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:33:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/13 13:33:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:33:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:33:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:33:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:33:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 53
2021/12/13 13:33:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1756
2021/12/13 13:33:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4203
2021/12/13 13:33:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:33:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:33:46 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:33:51 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 13:33:56 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 13:34:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 13:34:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 13:34:06 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/13 13:34:06 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:34:06 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/13 13:34:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:34:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:34:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:34:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:34:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:34:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:34:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.77s)
=== RUN   TestWithUserXattrs
2021/12/13 13:35:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 13:35:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/13 13:35:14 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:35:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/13 13:35:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:35:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:35:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:35:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:35:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:35:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:35:31 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:35:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/13 13:35:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 13:35:43 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 13:35:48 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 13:35:53 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 13:35:53 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/13 13:35:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:36:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:36:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 13:36:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:36:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:36:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:36:12 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 13:36:17 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 13:36:22 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 13:36:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 13:36:27 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/13 13:36:27 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:36:27 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/13 13:36:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:36:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:36:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:36:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:36:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:36:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:36:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (110.82s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/13 13:37:05 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:37:05 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 13:37:05 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:37:05 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/13 13:37:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:37:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:37:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:37:18 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:37:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/13 13:37:24 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 13:37:29 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 13:37:34 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 13:37:39 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 13:37:39 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/13 13:37:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:37:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:37:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:37:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 13:37:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:37:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:38:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:38:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4391
2021/12/13 13:38:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1891
2021/12/13 13:38:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:38:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:38:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2626
2021/12/13 13:38:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2626
2021/12/13 13:38:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:38:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:39:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16970
2021/12/13 13:39:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16970
2021/12/13 13:39:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 13:39:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:39:05 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:39:10 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:39:15 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:39:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:39:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:39:25 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 13:39:25 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:39:30 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 13:39:30 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 13:39:30 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:39:30 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/13 13:39:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:39:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 13:39:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 13:39:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:39:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:39:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 15636
2021/12/13 13:40:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3136
2021/12/13 13:40:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (187.66s)
=== RUN   TestMetaBucketDelete
2021/12/13 13:40:13 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 13:40:13 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 13:40:13 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:40:13 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/13 13:40:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:40:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:40:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:40:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:40:26 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:40:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 13:40:32 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 13:40:37 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 13:40:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 13:40:47 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 13:40:47 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/13 13:40:47 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/13 13:40:48 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/13 13:40:53 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 13:40:53 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 13:41:08 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 13:41:08 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 13:41:08 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:41:09 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/13 13:41:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:41:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:41:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:41:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.76s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/13 13:41:32 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:41:32 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:41:32 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:41:32 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 13:41:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:41:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:41:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:41:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:41:45 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:41:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 13:42:05 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/13 13:42:05 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/13 13:42:20 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:42:20 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:42:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:42:35 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:42:35 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:42:35 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 13:42:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:42:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:42:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:42:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.76s)
=== RUN   TestSourceBucketDelete
2021/12/13 13:42:58 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 13:42:58 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 13:42:58 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:42:58 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/13 13:43:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:43:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:43:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:43:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:43:11 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:43:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 13:43:18 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 13:43:23 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 13:43:28 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 13:43:33 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 13:43:33 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/13 13:43:33 handler_test.go:945: Deleting source bucket: default
2021/12/13 13:43:34 handler_test.go:947: Deleted source bucket: default
2021/12/13 13:43:39 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 13:43:39 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 13:43:55 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 13:43:55 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 13:43:55 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:43:55 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/13 13:43:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:44:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:44:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:44:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.29s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/13 13:44:18 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:44:18 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:44:18 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:44:18 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 13:44:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:44:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:44:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:44:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:44:31 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:44:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 13:44:51 handler_test.go:966: Deleting source bucket: default
2021/12/13 13:44:52 handler_test.go:968: Deleted source bucket: default
2021/12/13 13:45:07 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:45:07 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:45:22 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:45:22 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:45:22 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:45:22 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 13:45:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:45:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:45:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:45:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.76s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/13 13:45:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:45:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:45:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:45:45 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 13:45:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:45:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:45:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:45:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:45:59 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:46:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 13:46:18 handler_test.go:988: Deleting source bucket: default
2021/12/13 13:46:20 handler_test.go:990: Deleted source bucket: default
2021/12/13 13:46:20 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/13 13:46:20 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/13 13:46:35 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:46:35 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:46:51 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 13:46:51 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 13:46:51 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:46:51 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 13:46:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:46:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:46:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:47:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.21s)
=== RUN   TestUndeployDuringBootstrap
2021/12/13 13:47:14 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 13:47:14 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 13:47:14 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:47:14 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/13 13:47:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:47:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:47:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:47:27 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:47:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 13:47:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:47:40 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:47:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:47:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 13:47:40 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/13 13:47:40 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 13:47:45 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 13:47:50 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/13 13:47:55 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 13:48:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 13:48:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 13:48:05 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 13:48:05 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:48:30 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 13:48:30 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 13:48:30 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:48:30 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/13 13:48:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:48:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:48:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:48:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:48:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:48:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2911
2021/12/13 13:49:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 411
2021/12/13 13:49:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.53s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/13 13:49:08 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:49:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/13 13:49:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 13:49:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 13:49:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 13:49:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 13:49:29 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/13 13:49:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/13 13:49:29 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:49:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/13 13:49:35 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/13 13:49:35 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/13 13:49:35 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:49:35 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/13 13:49:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:49:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:49:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:49:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.59s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/13 13:49:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:49:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 13:49:58 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:49:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/13 13:49:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:50:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:50:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:50:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:50:11 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:50:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/13 13:50:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 13:50:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 13:50:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 13:50:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 13:50:32 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/13 13:50:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:51:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:51:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:51:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:51:18 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:51:23 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:51:23 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 13:51:23 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:53:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 13:53:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 13:53:08 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:53:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/13 13:53:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:53:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:53:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:53:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:53:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:53:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:53:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.07s)
=== RUN   TestUndeployWithKVFailover
2021/12/13 13:53:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 13:53:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/13 13:53:46 eventingSetup.go:46: No eventing-con process running
2021/12/13 13:53:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/13 13:53:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:53:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:53:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:53:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:53:59 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:54:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/13 13:54:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 13:54:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:54:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:54:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:54:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 13:54:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1
2021/12/13 13:54:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1
2021/12/13 13:54:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3413
2021/12/13 13:54:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3413
2021/12/13 13:54:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 13:54:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 13:54:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 13:54:47 Rebalance progress: 29.16666666666666
2021/12/13 13:54:52 Rebalance progress: 29.16666666666666
2021/12/13 13:54:57 Rebalance progress: 100
2021/12/13 13:54:57 Failing over: [127.0.0.1:19003]
2021/12/13 13:55:07 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 13:56:07 Removing node(s): [] from the cluster
2021/12/13 13:56:13 Rebalance progress: 29.16666666666666
2021/12/13 13:56:18 Rebalance progress: 29.16666666666666
2021/12/13 13:56:23 Rebalance progress: 100
2021/12/13 13:56:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 13:56:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/13 13:56:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:56:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 13:56:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/13 13:56:33 No eventing-con process running
2021/12/13 13:56:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/13 13:56:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:56:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:56:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 13:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:56:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:57:01 Waiting for dst bucket item count to get to: 0 curr count: 4266
2021/12/13 13:57:06 Waiting for dst bucket item count to get to: 0 curr count: 4266
2021/12/13 13:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (209.88s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/13 13:57:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 13:57:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/13 13:57:16 No eventing-con process running
2021/12/13 13:57:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/13 13:57:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:57:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:57:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:57:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:57:29 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/13 13:57:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/13 13:57:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 13:57:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 13:57:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 13:57:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 13:57:50 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/13 13:57:50 Starting bucket ops to source bucket
2021/12/13 13:58:11 Waiting for dst bucket item count to get to: 20000 curr count: 10549
2021/12/13 13:58:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 13:58:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:58:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:58:23 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/13 13:58:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 13:58:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:58:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:58:40 Waiting for dst bucket item count to get to: 0 curr count: 11341
2021/12/13 13:58:45 Waiting for dst bucket item count to get to: 0 curr count: 11341
2021/12/13 13:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:58:45 Starting bucket ops to source bucket
2021/12/13 13:58:58 Waiting for dst bucket item count to get to: 10000 curr count: 4341
2021/12/13 13:59:03 Waiting for dst bucket item count to get to: 10000 curr count: 4341
2021/12/13 13:59:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 13:59:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 13:59:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 13:59:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 13:59:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 13:59:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 13:59:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 13:59:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/13 13:59:23 No eventing-con process running
2021/12/13 13:59:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/13 13:59:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:59:34 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 13:59:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 13:59:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 13:59:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 13:59:51 Waiting for dst bucket item count to get to: 0 curr count: 9510
2021/12/13 13:59:56 Waiting for dst bucket item count to get to: 0 curr count: 9510
2021/12/13 13:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.21s)
=== RUN   TestTimerOverwrite
2021/12/13 14:00:06 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 14:00:06 App: TestTimerOverwrite got un-deployed
2021/12/13 14:00:06 No eventing-con process running
2021/12/13 14:00:06 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/13 14:00:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:00:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:00:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:00:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:00:19 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:00:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/13 14:00:25 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 14:00:30 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 14:00:35 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 14:00:40 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 14:00:40 App: TestTimerOverwrite got deployed
2021/12/13 14:00:50 Starting bucket ops to source bucket
2021/12/13 14:00:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:01:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:02:00 Waiting for dst bucket item count to get to: 5000 curr count: 4843
2021/12/13 14:02:05 Waiting for dst bucket item count to get to: 5000 curr count: 4843
2021/12/13 14:02:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 14:02:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:02:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:02:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 14:02:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 14:02:30 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 14:02:35 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 14:02:35 App: TestTimerOverwrite got un-deployed
2021/12/13 14:02:35 No eventing-con process running
2021/12/13 14:02:35 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/13 14:02:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:02:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 14:02:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:02:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:02:58 Waiting for dst bucket item count to get to: 0 curr count: 2982
2021/12/13 14:03:03 Waiting for dst bucket item count to get to: 0 curr count: 459
2021/12/13 14:03:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.85s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/13 14:03:08 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 14:03:09 Removing node(s): [] from the cluster
2021/12/13 14:03:14 Rebalance progress: 100
2021/12/13 14:03:14 Failing over: [127.0.0.1:19003]
2021/12/13 14:03:19 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 14:03:19 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 14:03:19 No eventing-con process running
2021/12/13 14:03:19 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/13 14:03:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:03:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:03:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:03:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:03:32 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:03:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 14:03:33 Starting bucket ops to source bucket
2021/12/13 14:03:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:03:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:03:43 Context: Post to set retry counter, response code: 200 dump: 
2021/12/13 14:03:43 set retry counter, response &{[] }
2021/12/13 14:04:48 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 14:04:48 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 14:04:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:04:48 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/13 14:04:53 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 14:04:53 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 14:04:53 No eventing-con process running
2021/12/13 14:04:53 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/13 14:04:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:05:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 14:05:09 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 14:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:05:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:05:21 Waiting for dst bucket item count to get to: 0 curr count: 2707
2021/12/13 14:05:26 Waiting for dst bucket item count to get to: 0 curr count: 2707
2021/12/13 14:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:05:26 Removing node(s): [] from the cluster
2021/12/13 14:05:31 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (148.35s)
=== RUN   TestOnUpdateSrcMutation
2021/12/13 14:05:41 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 14:05:41 App: TestOnUpdateSrcMutation got un-deployed
2021/12/13 14:05:41 No eventing-con process running
2021/12/13 14:05:41 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/13 14:05:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:05:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:05:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:05:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:05:54 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:05:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/13 14:05:56 Starting bucket ops to source bucket
2021/12/13 14:06:22 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 14:06:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:06:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:06:28 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 14:06:33 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 14:06:38 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 14:06:43 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 14:06:43 App: TestOnUpdateSrcMutation got un-deployed
2021/12/13 14:06:43 No eventing-con process running
2021/12/13 14:06:43 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/13 14:06:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:06:54 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 14:06:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 14:06:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:07:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.59s)
=== RUN   TestOnDeleteSrcMutation
2021/12/13 14:07:16 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 14:07:16 App: TestOnDeleteSrcMutation got un-deployed
2021/12/13 14:07:16 No eventing-con process running
2021/12/13 14:07:16 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/13 14:07:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:07:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:07:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:07:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:07:29 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:07:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/13 14:07:30 Starting bucket ops to source bucket
2021/12/13 14:07:59 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 14:07:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:07:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:08:04 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 14:08:09 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 14:08:14 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 14:08:19 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 14:08:19 App: TestOnDeleteSrcMutation got un-deployed
2021/12/13 14:08:19 No eventing-con process running
2021/12/13 14:08:19 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/13 14:08:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:08:30 Waiting for dst bucket item count to get to: 0 curr count: 3768
2021/12/13 14:08:35 Waiting for dst bucket item count to get to: 0 curr count: 3768
2021/12/13 14:08:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:08:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:08:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.27s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/13 14:08:52 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 14:08:52 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/13 14:08:52 No eventing-con process running
2021/12/13 14:08:52 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/13 14:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:08:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:09:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:09:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:09:05 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:09:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 14:09:06 Starting bucket ops to source bucket
2021/12/13 14:09:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 14:09:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:09:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:09:48 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 14:09:53 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 14:09:58 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 14:10:03 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 14:10:03 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/13 14:10:03 No eventing-con process running
2021/12/13 14:10:03 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/13 14:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:10:15 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 14:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.51s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/13 14:10:31 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 14:10:31 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/13 14:10:31 No eventing-con process running
2021/12/13 14:10:31 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/13 14:10:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:10:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:10:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:10:45 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:10:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 14:10:46 Starting bucket ops to source bucket
2021/12/13 14:11:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 14:11:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:11:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:11:40 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 14:11:45 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 14:11:50 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 14:11:55 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 14:11:55 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/13 14:11:55 No eventing-con process running
2021/12/13 14:11:55 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/13 14:11:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 14:12:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:12:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.42s)
=== RUN   TestInterHandlerRecursion
2021/12/13 14:12:23 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:12:23 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 14:12:23 No eventing-con process running
2021/12/13 14:12:23 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/13 14:12:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:12:31 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:12:41 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:12:41 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 14:12:41 No eventing-con process running
2021/12/13 14:12:41 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 14:12:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:12:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:12:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:12:54 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:12:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/13 14:12:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:13:00 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:13:05 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:13:10 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:13:15 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:13:15 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/13 14:13:15 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:13:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/13 14:13:15 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/13 14:13:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:13:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:13:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:13:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:13:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:13:35 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 14:13:35 No eventing-con process running
2021/12/13 14:13:35 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/13 14:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:13:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:13:53 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:13:53 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 14:13:53 No eventing-con process running
2021/12/13 14:13:53 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 14:13:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:14:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:14:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:14:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.72s)
=== RUN   TestInterBucketRecursion
2021/12/13 14:14:12 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:14:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:14:13 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:14:13 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:14:13 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/13 14:14:18 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 14:14:23 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 14:14:28 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 14:14:33 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 14:14:33 App: TestInterBucketRecursion_function1 got deployed
2021/12/13 14:14:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:14:38 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:14:43 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:14:48 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:14:53 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:14:53 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/13 14:14:53 No eventing-con process running
2021/12/13 14:14:53 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/13 14:14:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:14:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:15:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:15:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:15:11 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/13 14:15:11 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/13 14:15:11 No eventing-con process running
2021/12/13 14:15:11 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/13 14:15:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:15:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:15:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:15:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.55s)
=== RUN   TestLargeHandler
2021/12/13 14:15:24 Changed force_compress value to false
2021/12/13 14:15:24 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:15:24 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/13 14:15:24 Changed force_compress value to true
2021/12/13 14:15:24 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:15:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/13 14:15:31 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 14:15:36 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 14:15:41 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 14:15:46 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 14:15:46 App: TestLargeHandler_function got deployed
2021/12/13 14:15:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:15:51 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 14:15:56 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 14:16:01 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 14:16:06 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 14:16:06 App: TestLargeHandler_function got un-deployed
2021/12/13 14:16:06 No eventing-con process running
2021/12/13 14:16:07 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/13 14:16:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.32s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/13 14:16:29 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:16:30 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 14:16:30 No eventing-con process running
2021/12/13 14:16:30 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/13 14:16:31 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:16:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:16:48 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:16:48 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 14:16:48 No eventing-con process running
2021/12/13 14:16:48 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 14:16:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:16:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:16:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:17:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:17:01 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:17:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/13 14:17:02 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:17:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:17:12 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:17:17 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:17:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 14:17:22 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/13 14:17:22 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:17:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/13 14:17:23 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:17:28 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 14:17:33 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 14:17:38 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 14:17:43 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 14:17:43 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/13 14:17:43 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/13 14:17:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:17:48 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:17:53 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:17:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:18:03 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 14:18:03 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 14:18:03 No eventing-con process running
2021/12/13 14:18:04 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/13 14:18:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:18:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:18:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:18:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:18:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:18:23 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:18:28 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:18:33 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:18:38 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 14:18:38 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 14:18:38 No eventing-con process running
2021/12/13 14:18:38 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/13 14:18:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:18:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.36s)
=== RUN   TestAllowInterBucketRecursion
2021/12/13 14:18:56 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:18:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:18:57 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:19:02 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 14:19:07 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 14:19:12 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 14:19:17 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 14:19:17 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/13 14:19:17 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:19:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 14:19:18 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:19:23 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 14:19:28 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 14:19:33 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 14:19:38 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 14:19:38 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/13 14:19:38 TestAllowInterBucketRecursion_function2 is deployed
2021/12/13 14:19:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:19:43 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:19:48 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:19:53 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:19:58 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 14:19:58 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/13 14:19:58 No eventing-con process running
2021/12/13 14:19:58 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/13 14:20:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:20:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:20:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:20:18 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 14:20:23 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 14:20:28 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 14:20:33 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 14:20:33 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/13 14:20:33 No eventing-con process running
2021/12/13 14:20:33 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/13 14:20:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:20:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:20:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.90s)
=== RUN   TestInterBucketRecursion2
2021/12/13 14:20:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:20:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:21:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:21:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:21:15 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:21:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:21:21 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 14:21:26 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 14:21:31 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 14:21:36 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 14:21:36 App: TestInterBucketRecursion2_function1 got deployed
2021/12/13 14:21:36 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:21:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 14:21:42 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 14:21:47 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 14:21:52 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 14:21:57 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 14:21:57 App: TestInterBucketRecursion2_function2 got deployed
2021/12/13 14:21:57 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:21:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 14:22:04 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 14:22:09 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 14:22:14 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 14:22:19 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 14:22:19 App: TestInterBucketRecursion2_function3 got deployed
2021/12/13 14:22:19 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:22:19 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/13 14:22:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:22:24 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 14:22:29 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 14:22:34 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 14:22:39 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 14:22:39 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/13 14:22:39 No eventing-con process running
2021/12/13 14:22:39 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/13 14:22:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:22:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:22:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:22:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:22:57 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 14:23:02 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 14:23:07 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 14:23:12 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 14:23:12 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/13 14:23:12 No eventing-con process running
2021/12/13 14:23:12 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/13 14:23:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:23:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:23:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:23:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:23:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:23:31 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 14:23:36 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 14:23:41 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 14:23:46 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 14:23:46 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/13 14:23:46 No eventing-con process running
2021/12/13 14:23:46 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/13 14:23:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:23:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:23:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:24:04 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/13 14:24:04 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/13 14:24:04 No eventing-con process running
2021/12/13 14:24:04 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/13 14:24:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:24:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:24:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.48s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/13 14:24:24 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 14:24:29 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/13 14:24:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:24:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:25:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:25:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:25:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:25:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:26:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:26:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:26:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:26:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:27:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:27:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:27:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:27:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:28:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:28:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:28:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:28:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:29:02 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:29:17 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:29:32 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 14:29:47 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/13 14:30:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:30:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 14:30:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 14:30:08 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/13 14:30:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:30:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 14:30:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 14:30:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/13 14:30:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:30:18 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/13 14:30:18 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/13 14:30:18 eventingSetup.go:46: No eventing-con process running
2021/12/13 14:30:18 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/13 14:30:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:30:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:30:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:30:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:30:36 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:30:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 14:30:43 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 14:30:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 14:30:53 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 14:30:58 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 14:30:58 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 765e173bddfa85df96c0f24f323a8118
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 4daac716440afe4f61f86e37322a59f4
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: b5363cb4026a70b75c32112801f4038c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/13 14:30:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:30:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 14:31:03 Rebalance progress: 29.16666666666666
2021/12/13 14:31:08 Rebalance progress: 29.16666666666666
2021/12/13 14:31:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 765e173bddfa85df96c0f24f323a8118
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 4daac716440afe4f61f86e37322a59f4
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: b5363cb4026a70b75c32112801f4038c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/13 14:31:13 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:31:13 handler_test.go:1674: Deleting source bucket: default
2021/12/13 14:31:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 14:31:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 14:31:20 Rebalance progress: 29.16666666666666
2021/12/13 14:31:25 Rebalance progress: 29.16666666666666
2021/12/13 14:31:30 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 905
	vb list dump: [0-16, 18-25, 27-28, 30-46, 48-52, 54-55, 57-60, 62-69, 71-80, 82-86, 88-92, 94, 96-97, 99-124, 126-134, 136-139, 141, 143-147, 150, 152-158, 160-164, 166-176, 178-195, 197-201, 203-214, 216-221, 223-235, 237-264, 266-268, 270-282, 285, 287, 289-290, 292-297, 299-314, 316, 318-335, 337-340, 342, 344, 346-350, 352-368, 370-372, 374-389, 391-395, 397-401, 403-413, 415-417, 419-423, 426-451, 453, 455, 457, 459-466, 468-472, 476, 478-480, 484-487, 489-495, 497-501, 503-507, 509-582, 584-585, 587-616, 618-622, 624-633, 636-638, 640-649, 651-652, 654-659, 661-674, 676-677, 679-688, 690, 692-699, 701-714, 717, 719-723, 725-740, 742-746, 748, 750, 752-766, 768-774, 776-790, 792-800, 802-807, 809-810, 813-821, 823, 826, 828-836, 838-846, 848-873, 875-898, 900-902, 904-921, 923-936, 938-970, 972-978, 981, 983, 985-991, 993-998, 1000-1017, 1020, 1022-1023]
	stream status: stopped
	len: 119
	vb list dump: [17, 26, 29, 47, 53, 56, 61, 70, 81, 87, 93, 95, 98, 125, 135, 140, 142, 148-149, 151, 159, 165, 177, 196, 202, 215, 222, 236, 265, 269, 283-284, 286, 288, 291, 298, 315, 317, 336, 341, 343, 345, 351, 369, 373, 390, 396, 402, 414, 418, 424-425, 452, 454, 456, 458, 467, 473-475, 477, 481-483, 488, 496, 502, 508, 583, 586, 617, 623, 634-635, 639, 650, 653, 660, 675, 678, 689, 691, 700, 715-716, 718, 724, 741, 747, 749, 751, 767, 775, 791, 801, 808, 811-812, 822, 824-825, 827, 837, 847, 874, 899, 903, 922, 937, 971, 979-980, 982, 984, 992, 999, 1018-1019, 1021]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 765e173bddfa85df96c0f24f323a8118
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [114-124, 126-134, 136-139, 141, 143-147, 150, 152-158, 160-164, 166-176, 178-195, 197-201, 203-214, 216-221, 223-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-235, 237-264, 266-268, 270-282, 285, 287, 289-290, 292-297, 299-314, 316, 318-335, 337-340]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-16, 18-25, 27-28, 30-46, 48-52, 54-55, 57-60, 62-69, 71-80, 82-86, 88-92, 94, 96-97, 99-113]
Producer node: 172.31.2.134:9300	Node UUID: 4daac716440afe4f61f86e37322a59f4
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-921, 923-936, 938-970, 972-978, 981, 983, 985-991, 993-998, 1000-1017, 1020, 1022-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [683-688, 690, 692-699, 701-714, 717, 719-723, 725-740, 742-746, 748, 750, 752-766, 768-774, 776-790, 792-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-800, 802-807, 809-810, 813-821, 823, 826, 828-836, 838-846, 848-873, 875-898, 900-902, 904-910]
Producer node: 127.0.0.1:9303	Node UUID: b5363cb4026a70b75c32112801f4038c
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342, 344, 346-350, 352-368, 370-372, 374-389, 391-395, 397-401, 403-413, 415-417, 419-423, 426-451, 453, 455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [457, 459-466, 468-472, 476, 478-480, 484-487, 489-495, 497-501, 503-507, 509-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-582, 584-585, 587-616, 618-622, 624-633, 636-638, 640-649, 651-652, 654-659, 661-674, 676-677, 679-682]


2021/12/13 14:31:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:31:35 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/13 14:31:35 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/13 14:31:40 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/13 14:31:40 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/13 14:31:40 eventingSetup.go:46: No eventing-con process running
2021/12/13 14:31:40 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/13 14:31:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:31:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:31:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:31:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:31:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 14:31:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 14:31:59 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (462.30s)
=== RUN   TestN1QLRecursion
2021/12/13 14:32:02 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 14:32:02 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 14:32:02 Removing node(s): [] from the cluster
2021/12/13 14:32:07 Rebalance progress: 100
2021/12/13 14:32:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:32:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:32:30 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:32:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:32:31 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:32:36 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 14:32:41 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 14:32:46 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 14:32:51 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 14:32:51 App: TestN1QLRecursion_function1 got deployed
2021/12/13 14:32:51 TestN1QLRecursion_function1 is deployed
2021/12/13 14:32:51 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:32:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 14:32:52 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:32:57 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 14:33:02 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 14:33:07 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 14:33:12 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 14:33:12 App: TestN1QLRecursion_function3 got deployed
2021/12/13 14:33:12 TestN1QLRecursion_function3 is deployed
2021/12/13 14:33:12 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:33:12 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/13 14:33:12 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/13 14:33:12 Success: TestN1QLRecursion_function2 is not deployed
2021/12/13 14:33:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:33:17 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 14:33:22 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 14:33:27 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 14:33:32 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 14:33:32 App: TestN1QLRecursion_function1 got un-deployed
2021/12/13 14:33:32 No eventing-con process running
2021/12/13 14:33:33 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/13 14:33:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:33:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:33:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:33:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:33:51 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 14:33:56 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 14:34:01 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 14:34:06 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 14:34:06 App: TestN1QLRecursion_function3 got un-deployed
2021/12/13 14:34:06 No eventing-con process running
2021/12/13 14:34:06 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/13 14:34:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:34:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:34:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:34:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:34:21 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 14:34:26 Rebalance progress: 100
--- PASS: TestN1QLRecursion (144.96s)
=== RUN   TestN1QLAllowRecursion
2021/12/13 14:34:26 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 14:34:27 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 14:34:27 Removing node(s): [] from the cluster
2021/12/13 14:34:32 Rebalance progress: 100
2021/12/13 14:34:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:34:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:34:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:34:55 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:34:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:34:56 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:35:01 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 14:35:06 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 14:35:11 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 14:35:16 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 14:35:16 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/13 14:35:16 TestN1QLAllowRecursion_function1 is deployed
2021/12/13 14:35:16 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:35:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 14:35:17 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:35:22 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 14:35:27 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 14:35:32 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 14:35:37 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 14:35:37 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/13 14:35:37 TestN1QLAllowRecursion_function3 is deployed
2021/12/13 14:35:37 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:35:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 14:35:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/13 14:35:44 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 14:35:49 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 14:35:54 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 14:35:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 14:35:59 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/13 14:35:59 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/13 14:35:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:36:04 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 14:36:09 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 14:36:14 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 14:36:19 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 14:36:19 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/13 14:36:19 No eventing-con process running
2021/12/13 14:36:19 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/13 14:36:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:36:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:36:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:36:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:36:38 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 14:36:43 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 14:36:48 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 14:36:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 14:36:53 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/13 14:36:53 No eventing-con process running
2021/12/13 14:36:53 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/13 14:36:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:37:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:37:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:37:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:37:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:37:12 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 14:37:17 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 14:37:22 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 14:37:27 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 14:37:27 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/13 14:37:27 No eventing-con process running
2021/12/13 14:37:27 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/13 14:37:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:37:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:37:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:37:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:37:42 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 14:37:47 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.53s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/13 14:37:47 Sleeping for some time
2021/12/13 14:37:57 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 14:37:57 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/13 14:37:57 No eventing-con process running
2021/12/13 14:37:57 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/13 14:37:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:38:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:38:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:38:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:38:11 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:38:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 14:38:12 Starting bucket ops to source bucket
2021/12/13 14:39:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:39:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 14:39:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 14:39:06 Starting bucket ops to source bucket
2021/12/13 14:39:07 Starting bucket ops to source bucket
2021/12/13 14:40:13 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/13 14:40:18 Waiting for dst bucket item count to get to: 7000 curr count: 6221
2021/12/13 14:40:23 Waiting for dst bucket item count to get to: 7000 curr count: 6221
2021/12/13 14:40:23 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/13 14:40:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:40:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:40:28 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 14:40:33 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 14:40:38 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 14:40:43 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 14:40:43 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/13 14:40:43 No eventing-con process running
2021/12/13 14:40:43 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/13 14:40:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:40:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:40:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:41:07 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/13 14:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.77s)
=== RUN   TestTimerOverWriteSameReference
2021/12/13 14:41:12 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 14:41:12 App: TestTimerOverWriteSameReference got un-deployed
2021/12/13 14:41:12 No eventing-con process running
2021/12/13 14:41:12 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/13 14:41:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:41:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:41:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:41:25 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:41:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/13 14:41:32 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 14:41:37 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 14:41:42 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 14:41:47 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 14:41:47 App: TestTimerOverWriteSameReference got deployed
2021/12/13 14:41:57 Starting bucket ops to source bucket
2021/12/13 14:42:58 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/13 14:43:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:43:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:43:13 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 14:43:18 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 14:43:23 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 14:43:28 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 14:43:28 App: TestTimerOverWriteSameReference got un-deployed
2021/12/13 14:43:28 No eventing-con process running
2021/12/13 14:43:28 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/13 14:43:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:43:40 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:43:45 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:43:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:43:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:43:57 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 14:43:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.73s)
=== RUN   TestCancelTimerBucketop
2021/12/13 14:44:02 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 14:44:02 App: TestCancelTimerBucketop got un-deployed
2021/12/13 14:44:02 No eventing-con process running
2021/12/13 14:44:02 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/13 14:44:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:44:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:44:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:44:15 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:44:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/13 14:44:21 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 14:44:26 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 14:44:31 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 14:44:36 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 14:44:36 App: TestCancelTimerBucketop got deployed
2021/12/13 14:44:36 Starting bucket ops to source bucket
2021/12/13 14:44:44 Starting bucket ops to source bucket
2021/12/13 14:45:46 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 14:45:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:45:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:45:51 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 14:45:56 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 14:46:01 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 14:46:06 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 14:46:06 App: TestCancelTimerBucketop got un-deployed
2021/12/13 14:46:06 No eventing-con process running
2021/12/13 14:46:06 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/13 14:46:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:46:18 Waiting for dst bucket item count to get to: 0 curr count: 656
2021/12/13 14:46:23 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/13 14:46:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:46:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:46:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.43s)
=== RUN   TestJSExpiryDate
2021/12/13 14:46:40 Starting bucket ops to source bucket
2021/12/13 14:46:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 14:46:53 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/13 14:46:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/13 14:47:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:47:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:47:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:47:29 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 14:47:34 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 14:47:39 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 14:47:44 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 14:47:44 App: TestJSExpiryDate got un-deployed
2021/12/13 14:47:44 No eventing-con process running
2021/12/13 14:47:45 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/13 14:47:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:47:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:47:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:47:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.81s)
=== RUN   TestBinaryDoc
2021/12/13 14:48:04 Starting bucket ops to source bucket
2021/12/13 14:48:04 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:48:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/13 14:48:05 Starting bucket ops to source bucket
2021/12/13 14:48:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:48:05 Deleting source bucket docs now
2021/12/13 14:48:05 Starting bucket ops to source bucket
2021/12/13 14:48:42 Waiting for dst bucket item count to get to: 32 curr count: 23
2021/12/13 14:48:42 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/13 14:48:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:48:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:48:47 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 14:48:52 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 14:48:57 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 14:49:02 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 14:49:02 App: TestBinaryDoc got un-deployed
2021/12/13 14:49:02 No eventing-con process running
2021/12/13 14:49:02 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/13 14:49:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:49:14 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/13 14:49:19 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/13 14:49:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:49:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:49:31 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/13 14:49:36 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/13 14:49:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.25s)
=== RUN   TestConstantBindings
2021/12/13 14:49:36 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:49:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/13 14:49:37 Starting bucket ops to source bucket
2021/12/13 14:49:43 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 14:49:48 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 14:49:53 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 14:49:58 Waiting for dst bucket item count to get to: 6000 curr count: 647
2021/12/13 14:50:03 Waiting for dst bucket item count to get to: 6000 curr count: 647
2021/12/13 14:50:03 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/13 14:50:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:50:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:50:08 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 14:50:13 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 14:50:18 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 14:50:23 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 14:50:23 App: TestConstantBindings got un-deployed
2021/12/13 14:50:23 No eventing-con process running
2021/12/13 14:50:23 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/13 14:50:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:50:35 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:50:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:50:47 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/13 14:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.76s)
=== RUN   TestCodeInsight
2021/12/13 14:50:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 14:50:52 App: TestCodeInsight got un-deployed
2021/12/13 14:50:52 No eventing-con process running
2021/12/13 14:50:52 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/13 14:50:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:50:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:51:05 Starting bucket ops to source bucket
2021/12/13 14:51:06 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:51:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/13 14:51:12 Waiting for app: TestCodeInsight to get deployed
2021/12/13 14:51:17 Waiting for app: TestCodeInsight to get deployed
2021/12/13 14:51:22 Waiting for app: TestCodeInsight to get deployed
2021/12/13 14:51:27 Waiting for app: TestCodeInsight to get deployed
2021/12/13 14:51:27 App: TestCodeInsight got deployed
2021/12/13 14:51:32 Waiting for dst bucket item count to get to: 1000 curr count: 227
2021/12/13 14:51:32 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 14:51:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:51:37 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 14:51:42 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 14:51:47 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 14:51:52 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 14:51:52 App: TestCodeInsight got un-deployed
2021/12/13 14:51:52 No eventing-con process running
2021/12/13 14:51:53 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/13 14:51:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:52:04 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:52:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (89.40s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12920.667s

Functional tests took 3 hours, 35 minutes and 24 seconds.

N1ql tests

2021/12/13 14:52:26 Started n1ql
2021/12/13 14:52:36 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 14:52:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 14:52:36 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 14:52:37 Removing node(s): [] from the cluster
2021/12/13 14:52:42 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/13 14:52:47 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/13 14:52:47 App: TestRecursiveMutationN1QL got un-deployed
2021/12/13 14:52:47 No eventing-con process running
2021/12/13 14:52:47 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/13 14:52:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:52:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:52:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:53:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:53:00 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:53:00 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.52s)
=== RUN   TestFlexReset1
2021/12/13 14:53:05 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 14:53:05 App: TestFlexReset1 got un-deployed
2021/12/13 14:53:05 No eventing-con process running
2021/12/13 14:53:05 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/13 14:53:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:53:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:53:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:53:19 Starting bucket ops to source bucket
2021/12/13 14:53:20 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:53:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/13 14:53:26 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 14:53:31 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 14:53:36 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 14:53:41 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 14:53:41 App: TestFlexReset1 got deployed
2021/12/13 14:53:41 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 14:53:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:53:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:53:46 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 14:53:51 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 14:53:56 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 14:54:01 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 14:54:01 App: TestFlexReset1 got un-deployed
2021/12/13 14:54:01 No eventing-con process running
2021/12/13 14:54:01 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/13 14:54:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:54:13 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:54:18 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:54:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:54:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:54:30 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:54:35 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.21s)
=== RUN   TestFlexReset2
2021/12/13 14:54:40 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 14:54:40 App: TestFlexReset2 got un-deployed
2021/12/13 14:54:40 No eventing-con process running
2021/12/13 14:54:40 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/13 14:54:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:54:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:54:53 Starting bucket ops to source bucket
2021/12/13 14:54:54 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:54:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/13 14:55:00 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 14:55:05 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 14:55:10 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 14:55:15 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 14:55:15 App: TestFlexReset2 got deployed
2021/12/13 14:55:20 Waiting for dst bucket item count to get to: 1000 curr count: 392
2021/12/13 14:55:25 Waiting for dst bucket item count to get to: 1000 curr count: 847
2021/12/13 14:55:25 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 14:55:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:55:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:55:30 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 14:55:35 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 14:55:40 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 14:55:45 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 14:55:45 App: TestFlexReset2 got un-deployed
2021/12/13 14:55:45 No eventing-con process running
2021/12/13 14:55:45 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/13 14:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:55:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:55:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:56:09 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 14:56:14 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 14:56:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.25s)
=== RUN   TestN1QLLabelledBreak
2021/12/13 14:56:19 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 14:56:19 App: TestN1QLLabelledBreak got un-deployed
2021/12/13 14:56:19 No eventing-con process running
2021/12/13 14:56:19 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/13 14:56:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:56:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:56:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:56:32 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:56:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/13 14:56:39 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 14:56:44 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 14:56:49 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 14:56:54 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 14:56:54 App: TestN1QLLabelledBreak got deployed
2021/12/13 14:56:54 Starting bucket ops to source bucket
2021/12/13 14:57:03 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/13 14:57:08 Waiting for dst bucket item count to get to: 10000 curr count: 249
2021/12/13 14:57:13 Waiting for dst bucket item count to get to: 10000 curr count: 793
2021/12/13 14:57:18 Waiting for dst bucket item count to get to: 10000 curr count: 2711
2021/12/13 14:57:23 Waiting for dst bucket item count to get to: 10000 curr count: 4761
2021/12/13 14:57:28 Waiting for dst bucket item count to get to: 10000 curr count: 6785
2021/12/13 14:57:33 Waiting for dst bucket item count to get to: 10000 curr count: 8778
2021/12/13 14:57:38 Waiting for dst bucket item count to get to: 10000 curr count: 9587
2021/12/13 14:57:38 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 14:57:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:57:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:57:43 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 14:57:48 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 14:57:53 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 14:57:58 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 14:57:58 App: TestN1QLLabelledBreak got un-deployed
2021/12/13 14:57:58 No eventing-con process running
2021/12/13 14:57:58 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/13 14:58:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:58:10 Waiting for dst bucket item count to get to: 0 curr count: 3243
2021/12/13 14:58:15 Waiting for dst bucket item count to get to: 0 curr count: 3243
2021/12/13 14:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:58:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:58:27 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 14:58:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (132.99s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/13 14:58:32 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 14:58:32 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/13 14:58:32 No eventing-con process running
2021/12/13 14:58:32 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/13 14:58:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:58:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:58:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 14:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 14:58:45 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/13 14:58:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/13 14:58:52 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 14:58:57 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 14:59:02 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 14:59:07 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 14:59:07 App: TestN1QLUnlabelledBreak got deployed
2021/12/13 14:59:07 Starting bucket ops to source bucket
2021/12/13 14:59:16 Waiting for dst bucket item count to get to: 10000 curr count: 299
2021/12/13 14:59:21 Waiting for dst bucket item count to get to: 10000 curr count: 299
2021/12/13 14:59:26 Waiting for dst bucket item count to get to: 10000 curr count: 299
2021/12/13 14:59:31 Waiting for dst bucket item count to get to: 10000 curr count: 4982
2021/12/13 14:59:36 Waiting for dst bucket item count to get to: 10000 curr count: 4982
2021/12/13 14:59:36 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 14:59:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 14:59:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 14:59:41 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 14:59:46 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 14:59:51 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 14:59:56 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 14:59:56 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/13 14:59:56 No eventing-con process running
2021/12/13 14:59:57 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/13 14:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:00:08 Waiting for dst bucket item count to get to: 0 curr count: 4046
2021/12/13 15:00:13 Waiting for dst bucket item count to get to: 0 curr count: 1546
2021/12/13 15:00:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:00:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:00:25 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 15:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.21s)
=== RUN   TestN1QLThrowStatement
2021/12/13 15:00:30 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 15:00:30 App: TestN1QLThrowStatement got un-deployed
2021/12/13 15:00:30 No eventing-con process running
2021/12/13 15:00:30 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/13 15:00:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:00:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:00:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:00:44 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:00:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/13 15:00:50 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 15:00:55 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 15:01:00 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 15:01:05 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 15:01:05 App: TestN1QLThrowStatement got deployed
2021/12/13 15:01:05 Starting bucket ops to source bucket
2021/12/13 15:01:14 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/13 15:01:19 Waiting for dst bucket item count to get to: 10000 curr count: 159
2021/12/13 15:01:24 Waiting for dst bucket item count to get to: 10000 curr count: 455
2021/12/13 15:01:29 Waiting for dst bucket item count to get to: 10000 curr count: 4109
2021/12/13 15:01:34 Waiting for dst bucket item count to get to: 10000 curr count: 4109
2021/12/13 15:01:39 Waiting for dst bucket item count to get to: 10000 curr count: 8160
2021/12/13 15:01:44 Waiting for dst bucket item count to get to: 10000 curr count: 8160
2021/12/13 15:01:44 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 15:01:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:01:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:01:49 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 15:01:55 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 15:02:00 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 15:02:05 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 15:02:05 App: TestN1QLThrowStatement got un-deployed
2021/12/13 15:02:05 No eventing-con process running
2021/12/13 15:02:05 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/13 15:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:02:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:02:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:02:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:02:28 Waiting for dst bucket item count to get to: 0 curr count: 7539
2021/12/13 15:02:33 Waiting for dst bucket item count to get to: 0 curr count: 2539
2021/12/13 15:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (128.07s)
=== RUN   TestN1QLNestedForLoop
2021/12/13 15:02:38 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 15:02:38 App: TestN1QLNestedForLoop got un-deployed
2021/12/13 15:02:38 No eventing-con process running
2021/12/13 15:02:38 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/13 15:02:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:02:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:02:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:02:52 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:02:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/13 15:02:58 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 15:03:03 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 15:03:08 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 15:03:13 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 15:03:13 App: TestN1QLNestedForLoop got deployed
2021/12/13 15:03:13 Starting bucket ops to source bucket
2021/12/13 15:03:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 15:03:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 15:03:33 Waiting for dst bucket item count to get to: 5000 curr count: 509
2021/12/13 15:03:38 Waiting for dst bucket item count to get to: 5000 curr count: 1638
2021/12/13 15:03:43 Waiting for dst bucket item count to get to: 5000 curr count: 2829
2021/12/13 15:03:48 Waiting for dst bucket item count to get to: 5000 curr count: 4015
2021/12/13 15:03:53 Waiting for dst bucket item count to get to: 5000 curr count: 4972
2021/12/13 15:03:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 15:03:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:03:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:03:58 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 15:04:03 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 15:04:08 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 15:04:13 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 15:04:13 App: TestN1QLNestedForLoop got un-deployed
2021/12/13 15:04:13 No eventing-con process running
2021/12/13 15:04:13 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/13 15:04:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:04:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:04:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:04:26 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:04:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (123.12s)
=== RUN   TestN1QLAutoClose
2021/12/13 15:04:41 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 15:04:41 App: TestN1QLAutoClose got un-deployed
2021/12/13 15:04:41 No eventing-con process running
2021/12/13 15:04:41 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/13 15:04:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:04:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:04:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:04:55 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:04:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/13 15:05:01 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 15:05:06 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 15:05:11 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 15:05:16 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 15:05:16 App: TestN1QLAutoClose got deployed
2021/12/13 15:05:16 Starting bucket ops to source bucket
2021/12/13 15:05:22 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/13 15:05:27 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/13 15:05:32 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/13 15:05:37 Waiting for dst bucket item count to get to: 500 curr count: 26
2021/12/13 15:05:42 Waiting for dst bucket item count to get to: 500 curr count: 26
2021/12/13 15:05:47 Waiting for dst bucket item count to get to: 500 curr count: 49
2021/12/13 15:05:52 Waiting for dst bucket item count to get to: 500 curr count: 49
2021/12/13 15:05:57 Waiting for dst bucket item count to get to: 500 curr count: 71
2021/12/13 15:06:02 Waiting for dst bucket item count to get to: 500 curr count: 71
2021/12/13 15:06:07 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/13 15:06:12 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/13 15:06:17 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/13 15:06:22 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/13 15:06:27 Waiting for dst bucket item count to get to: 500 curr count: 113
2021/12/13 15:06:32 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/13 15:06:37 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/13 15:06:42 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/13 15:06:47 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/13 15:06:52 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/13 15:06:57 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/13 15:07:02 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/13 15:07:07 Waiting for dst bucket item count to get to: 500 curr count: 210
2021/12/13 15:07:12 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/13 15:07:17 Waiting for dst bucket item count to get to: 500 curr count: 230
2021/12/13 15:07:22 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/13 15:07:27 Waiting for dst bucket item count to get to: 500 curr count: 248
2021/12/13 15:07:32 Waiting for dst bucket item count to get to: 500 curr count: 259
2021/12/13 15:07:37 Waiting for dst bucket item count to get to: 500 curr count: 270
2021/12/13 15:07:42 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/13 15:07:47 Waiting for dst bucket item count to get to: 500 curr count: 295
2021/12/13 15:07:52 Waiting for dst bucket item count to get to: 500 curr count: 304
2021/12/13 15:07:57 Waiting for dst bucket item count to get to: 500 curr count: 317
2021/12/13 15:08:02 Waiting for dst bucket item count to get to: 500 curr count: 325
2021/12/13 15:08:07 Waiting for dst bucket item count to get to: 500 curr count: 337
2021/12/13 15:08:12 Waiting for dst bucket item count to get to: 500 curr count: 347
2021/12/13 15:08:17 Waiting for dst bucket item count to get to: 500 curr count: 358
2021/12/13 15:08:22 Waiting for dst bucket item count to get to: 500 curr count: 367
2021/12/13 15:08:27 Waiting for dst bucket item count to get to: 500 curr count: 376
2021/12/13 15:08:32 Waiting for dst bucket item count to get to: 500 curr count: 388
2021/12/13 15:08:37 Waiting for dst bucket item count to get to: 500 curr count: 408
2021/12/13 15:08:42 Waiting for dst bucket item count to get to: 500 curr count: 408
2021/12/13 15:08:47 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/13 15:08:52 Waiting for dst bucket item count to get to: 500 curr count: 428
2021/12/13 15:08:57 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/13 15:09:02 Waiting for dst bucket item count to get to: 500 curr count: 450
2021/12/13 15:09:07 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/13 15:09:12 Waiting for dst bucket item count to get to: 500 curr count: 472
2021/12/13 15:09:17 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/13 15:09:22 Waiting for dst bucket item count to get to: 500 curr count: 492
2021/12/13 15:09:22 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/13 15:09:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:09:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:09:27 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 15:09:32 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 15:09:37 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 15:09:42 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 15:09:42 App: TestN1QLAutoClose got un-deployed
2021/12/13 15:09:42 No eventing-con process running
2021/12/13 15:09:42 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/13 15:09:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:09:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 15:09:59 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/13 15:09:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:10:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:10:11 Waiting for dst bucket item count to get to: 0 curr count: 480
2021/12/13 15:10:16 Waiting for dst bucket item count to get to: 0 curr count: 480
2021/12/13 15:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.63s)
=== RUN   TestN1QLPosParams
2021/12/13 15:10:21 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 15:10:21 App: TestN1QLPosParams got un-deployed
2021/12/13 15:10:21 No eventing-con process running
2021/12/13 15:10:21 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/13 15:10:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:10:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:10:35 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:10:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/13 15:10:41 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 15:10:46 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 15:10:51 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 15:10:56 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 15:10:56 App: TestN1QLPosParams got deployed
2021/12/13 15:10:56 Starting bucket ops to source bucket
2021/12/13 15:11:06 Waiting for dst bucket item count to get to: 5000 curr count: 785
2021/12/13 15:11:11 Waiting for dst bucket item count to get to: 5000 curr count: 785
2021/12/13 15:11:11 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 15:11:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:11:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:11:16 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 15:11:21 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 15:11:26 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 15:11:31 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 15:11:31 App: TestN1QLPosParams got un-deployed
2021/12/13 15:11:31 No eventing-con process running
2021/12/13 15:11:31 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/13 15:11:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:11:43 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:11:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 15:11:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:11:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:11:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:12:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 15:12:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.35s)
=== RUN   TestN1QLExhaustConnPool
2021/12/13 15:12:09 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 15:12:09 App: TestN1QLExhaustConnPool got un-deployed
2021/12/13 15:12:09 No eventing-con process running
2021/12/13 15:12:09 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/13 15:12:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:12:23 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/13 15:12:29 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 15:12:34 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 15:12:39 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 15:12:44 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 15:12:44 App: TestN1QLExhaustConnPool got deployed
2021/12/13 15:12:44 Starting bucket ops to source bucket
2021/12/13 15:12:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 15:12:54 Waiting for dst bucket item count to get to: 100 curr count: 47
2021/12/13 15:12:59 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/13 15:12:59 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 15:12:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:13:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:13:05 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 15:13:10 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 15:13:15 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 15:13:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 15:13:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/13 15:13:20 No eventing-con process running
2021/12/13 15:13:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/13 15:13:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:13:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 15:13:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 15:13:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:13:48 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/13 15:13:53 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/13 15:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (109.03s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/13 15:13:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 15:13:58 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/13 15:13:58 No eventing-con process running
2021/12/13 15:13:58 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/13 15:14:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:14:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:14:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:14:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:14:12 Starting bucket ops to source bucket
2021/12/13 15:14:29 Starting bucket ops to source bucket
2021/12/13 15:14:47 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15294
2021/12/13 15:14:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/13 15:14:53 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 15:14:58 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 15:15:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 15:15:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 15:15:08 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/13 15:15:08 Starting bucket ops to source bucket
2021/12/13 15:15:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:18 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:15:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:16:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 15:16:03 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/13 15:16:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:16:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:16:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 15:16:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 15:16:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 15:16:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 15:16:23 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/13 15:16:23 No eventing-con process running
2021/12/13 15:16:24 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/13 15:16:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:16:35 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/13 15:16:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:16:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:16:47 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/13 15:16:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (173.92s)
=== RUN   TestN1QLQueryConsistency
2021/12/13 15:16:52 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 15:16:52 App: n1qlquery_consistency got un-deployed
2021/12/13 15:16:52 No eventing-con process running
2021/12/13 15:16:52 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/13 15:16:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:16:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:17:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:17:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:17:06 Starting bucket ops to source bucket
2021/12/13 15:17:09 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:17:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/13 15:17:15 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 15:17:20 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 15:17:25 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 15:17:30 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 15:17:30 App: n1qlquery_consistency got deployed
2021/12/13 15:17:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 15:17:40 Waiting for dst bucket item count to get to: 5000 curr count: 3898
2021/12/13 15:17:45 Waiting for dst bucket item count to get to: 5000 curr count: 3898
2021/12/13 15:17:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 15:17:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 15:17:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:17:50 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 15:17:55 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 15:18:00 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 15:18:05 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 15:18:05 App: n1qlquery_consistency got un-deployed
2021/12/13 15:18:05 No eventing-con process running
2021/12/13 15:18:05 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/13 15:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:18:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:18:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 15:18:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 15:18:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 15:18:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.74s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1563.604s

N1ql tests took 0 hours, 26 minutes and 8 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/13 15:20:09 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:55046->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:55046->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/13 15:20:28 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/13 15:20:33 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 15:20:33 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/13 15:20:33 No eventing-con process running
2021/12/13 15:20:33 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/13 15:20:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:20:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:20:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:20:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:20:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:20:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:20:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:20:51 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:20:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 15:20:57 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 15:21:02 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 15:21:07 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 15:21:12 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 15:21:12 App: TestDuplicateWithEventingRebIn got deployed
2021/12/13 15:21:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:21:12 Starting bucket ops to source bucket
2021/12/13 15:21:27 Removing node(s): [] from the cluster
2021/12/13 15:21:32 Rebalance progress: 0
2021/12/13 15:21:37 Rebalance progress: 0
2021/12/13 15:21:42 Rebalance progress: 25
2021/12/13 15:21:47 Rebalance progress: 25
2021/12/13 15:21:52 Rebalance progress: 25
2021/12/13 15:21:57 Rebalance progress: 25
2021/12/13 15:22:02 Rebalance progress: 25
2021/12/13 15:22:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:22:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:22:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 15:22:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:22:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:22:12 Rebalance progress: 25
2021/12/13 15:22:17 Rebalance progress: 25
2021/12/13 15:22:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:22:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:22:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 15:22:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:22:22 Waiting for src bucket item count to get to: 200000 curr count: 93687
2021/12/13 15:22:27 Waiting for src bucket item count to get to: 200000 curr count: 110230
2021/12/13 15:22:32 Waiting for src bucket item count to get to: 200000 curr count: 110230
2021/12/13 15:22:37 Waiting for src bucket item count to get to: 200000 curr count: 126485
2021/12/13 15:22:42 Waiting for src bucket item count to get to: 200000 curr count: 126485
2021/12/13 15:22:47 Waiting for src bucket item count to get to: 200000 curr count: 143303
2021/12/13 15:22:52 Waiting for src bucket item count to get to: 200000 curr count: 143303
2021/12/13 15:22:57 Waiting for src bucket item count to get to: 200000 curr count: 159735
2021/12/13 15:23:02 Waiting for src bucket item count to get to: 200000 curr count: 159735
2021/12/13 15:23:07 Waiting for src bucket item count to get to: 200000 curr count: 176090
2021/12/13 15:23:12 Waiting for src bucket item count to get to: 200000 curr count: 176090
2021/12/13 15:23:17 Waiting for src bucket item count to get to: 200000 curr count: 192725
2021/12/13 15:23:22 Waiting for src bucket item count to get to: 200000 curr count: 192725
2021/12/13 15:23:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:23:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:23:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:23:32 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 15:23:37 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 15:23:42 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 15:23:47 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 15:23:47 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/13 15:23:47 No eventing-con process running
2021/12/13 15:23:47 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/13 15:23:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:23:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:23:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:23:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:24:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:24:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:24:11 Waiting for dst bucket item count to get to: 0 curr count: 177719
2021/12/13 15:24:16 Waiting for dst bucket item count to get to: 0 curr count: 177719
2021/12/13 15:24:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (238.29s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/13 15:24:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:24:21 Removing node(s): [] from the cluster
2021/12/13 15:24:26 Rebalance progress: 100
2021/12/13 15:24:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 15:24:31 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/13 15:24:31 No eventing-con process running
2021/12/13 15:24:31 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/13 15:24:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:24:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:24:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:24:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:24:50 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:24:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 15:24:56 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 15:25:01 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 15:25:06 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 15:25:11 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 15:25:11 App: TestDuplicateWithEventingRebOut got deployed
2021/12/13 15:25:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:25:11 Starting bucket ops to source bucket
2021/12/13 15:25:16 Rebalance progress: 25
2021/12/13 15:25:21 Rebalance progress: 25
2021/12/13 15:25:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:25:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:25:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:25:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 15:25:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:25:26 Waiting for src bucket item count to get to: 200000 curr count: 13141
2021/12/13 15:25:31 Waiting for src bucket item count to get to: 200000 curr count: 29309
2021/12/13 15:25:36 Waiting for src bucket item count to get to: 200000 curr count: 29309
2021/12/13 15:25:41 Waiting for src bucket item count to get to: 200000 curr count: 45883
2021/12/13 15:25:46 Waiting for src bucket item count to get to: 200000 curr count: 45883
2021/12/13 15:25:51 Waiting for src bucket item count to get to: 200000 curr count: 62555
2021/12/13 15:25:56 Waiting for src bucket item count to get to: 200000 curr count: 62555
2021/12/13 15:26:01 Waiting for src bucket item count to get to: 200000 curr count: 62555
2021/12/13 15:26:06 Waiting for src bucket item count to get to: 200000 curr count: 78461
2021/12/13 15:26:11 Waiting for src bucket item count to get to: 200000 curr count: 78461
2021/12/13 15:26:16 Waiting for src bucket item count to get to: 200000 curr count: 94555
2021/12/13 15:26:21 Waiting for src bucket item count to get to: 200000 curr count: 94555
2021/12/13 15:26:26 Waiting for src bucket item count to get to: 200000 curr count: 110816
2021/12/13 15:26:31 Waiting for src bucket item count to get to: 200000 curr count: 110816
2021/12/13 15:26:36 Waiting for src bucket item count to get to: 200000 curr count: 127131
2021/12/13 15:26:41 Waiting for src bucket item count to get to: 200000 curr count: 127131
2021/12/13 15:26:46 Waiting for src bucket item count to get to: 200000 curr count: 143527
2021/12/13 15:26:51 Waiting for src bucket item count to get to: 200000 curr count: 143527
2021/12/13 15:26:56 Waiting for src bucket item count to get to: 200000 curr count: 159871
2021/12/13 15:27:01 Waiting for src bucket item count to get to: 200000 curr count: 159871
2021/12/13 15:27:06 Waiting for src bucket item count to get to: 200000 curr count: 176197
2021/12/13 15:27:11 Waiting for src bucket item count to get to: 200000 curr count: 176197
2021/12/13 15:27:16 Waiting for src bucket item count to get to: 200000 curr count: 192350
2021/12/13 15:27:21 Waiting for src bucket item count to get to: 200000 curr count: 192350
2021/12/13 15:27:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:27:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:27:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:27:31 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 15:27:36 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 15:27:41 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 15:27:46 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 15:27:46 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/13 15:27:46 No eventing-con process running
2021/12/13 15:27:46 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/13 15:27:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:27:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:27:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:27:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:28:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:28:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:28:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.86s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/13 15:28:15 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 15:28:15 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/13 15:28:15 No eventing-con process running
2021/12/13 15:28:15 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/13 15:28:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:28:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:28:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:28:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:28:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:28:33 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:28:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 15:28:39 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 15:28:44 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 15:28:49 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 15:28:54 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 15:28:54 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/13 15:28:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:28:54 Starting bucket ops to source bucket
2021/12/13 15:28:55 Removing node(s): [] from the cluster
2021/12/13 15:29:00 Rebalance progress: 25
2021/12/13 15:29:05 Rebalance progress: 25
2021/12/13 15:29:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:29:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 15:29:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:29:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:29:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:29:22 Rebalance progress: 16.66666666666667
2021/12/13 15:29:27 Rebalance progress: 16.66666666666667
2021/12/13 15:29:32 Rebalance progress: 16.66666666666667
2021/12/13 15:29:37 Rebalance progress: 35.68232662192393
2021/12/13 15:29:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:29:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:29:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 15:29:47 Rebalance progress: 16.66666666666667
2021/12/13 15:29:52 Rebalance progress: 16.66666666666667
2021/12/13 15:29:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:29:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:29:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:29:57 Waiting for src bucket item count to get to: 200000 curr count: 82675
2021/12/13 15:30:02 Waiting for src bucket item count to get to: 200000 curr count: 98534
2021/12/13 15:30:07 Waiting for src bucket item count to get to: 200000 curr count: 98534
2021/12/13 15:30:12 Waiting for src bucket item count to get to: 200000 curr count: 115016
2021/12/13 15:30:17 Waiting for src bucket item count to get to: 200000 curr count: 115016
2021/12/13 15:30:22 Waiting for src bucket item count to get to: 200000 curr count: 131107
2021/12/13 15:30:27 Waiting for src bucket item count to get to: 200000 curr count: 131107
2021/12/13 15:30:32 Waiting for src bucket item count to get to: 200000 curr count: 147115
2021/12/13 15:30:37 Waiting for src bucket item count to get to: 200000 curr count: 147115
2021/12/13 15:30:42 Waiting for src bucket item count to get to: 200000 curr count: 163557
2021/12/13 15:30:47 Waiting for src bucket item count to get to: 200000 curr count: 163557
2021/12/13 15:30:52 Waiting for src bucket item count to get to: 200000 curr count: 179450
2021/12/13 15:30:57 Waiting for src bucket item count to get to: 200000 curr count: 179450
2021/12/13 15:31:02 Waiting for src bucket item count to get to: 200000 curr count: 195421
2021/12/13 15:31:07 Waiting for src bucket item count to get to: 200000 curr count: 195421
2021/12/13 15:31:12 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:31:12 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:31:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:31:17 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 15:31:23 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 15:31:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 15:31:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 15:31:33 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/13 15:31:33 No eventing-con process running
2021/12/13 15:31:33 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/13 15:31:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:31:44 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:31:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:31:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:31:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:31:56 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:31:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (231.31s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/13 15:32:06 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 15:32:06 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/13 15:32:06 No eventing-con process running
2021/12/13 15:32:06 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/13 15:32:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:32:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:32:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:32:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:32:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:32:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:32:25 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:32:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 15:32:31 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 15:32:36 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 15:32:41 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 15:32:46 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 15:32:46 App: TestDuplicateWithKVRebIn got deployed
2021/12/13 15:32:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 15:32:46 Starting bucket ops to source bucket
2021/12/13 15:32:46 Removing node(s): [] from the cluster
2021/12/13 15:32:51 Rebalance progress: 16.66666666666667
2021/12/13 15:32:56 Rebalance progress: 16.66666666666667
2021/12/13 15:33:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:33:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:33:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:33:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:33:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:33:06 Rebalance progress: 16.66666666666667
2021/12/13 15:33:11 Rebalance progress: 16.66666666666667
2021/12/13 15:33:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:33:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:33:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:33:17 Waiting for src bucket item count to get to: 200000 curr count: 37096
2021/12/13 15:33:22 Waiting for src bucket item count to get to: 200000 curr count: 53383
2021/12/13 15:33:27 Waiting for src bucket item count to get to: 200000 curr count: 53383
2021/12/13 15:33:32 Waiting for src bucket item count to get to: 200000 curr count: 69427
2021/12/13 15:33:37 Waiting for src bucket item count to get to: 200000 curr count: 69427
2021/12/13 15:33:42 Waiting for src bucket item count to get to: 200000 curr count: 85107
2021/12/13 15:33:47 Waiting for src bucket item count to get to: 200000 curr count: 85107
2021/12/13 15:33:52 Waiting for src bucket item count to get to: 200000 curr count: 101584
2021/12/13 15:33:57 Waiting for src bucket item count to get to: 200000 curr count: 101584
2021/12/13 15:34:02 Waiting for src bucket item count to get to: 200000 curr count: 118169
2021/12/13 15:34:07 Waiting for src bucket item count to get to: 200000 curr count: 118169
2021/12/13 15:34:12 Waiting for src bucket item count to get to: 200000 curr count: 134609
2021/12/13 15:34:17 Waiting for src bucket item count to get to: 200000 curr count: 134609
2021/12/13 15:34:22 Waiting for src bucket item count to get to: 200000 curr count: 134609
2021/12/13 15:34:27 Waiting for src bucket item count to get to: 200000 curr count: 150973
2021/12/13 15:34:32 Waiting for src bucket item count to get to: 200000 curr count: 150973
2021/12/13 15:34:37 Waiting for src bucket item count to get to: 200000 curr count: 167449
2021/12/13 15:34:42 Waiting for src bucket item count to get to: 200000 curr count: 167449
2021/12/13 15:34:47 Waiting for src bucket item count to get to: 200000 curr count: 182671
2021/12/13 15:34:52 Waiting for src bucket item count to get to: 200000 curr count: 182671
2021/12/13 15:34:57 Waiting for src bucket item count to get to: 200000 curr count: 199023
2021/12/13 15:35:02 Waiting for src bucket item count to get to: 200000 curr count: 199023
2021/12/13 15:35:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:35:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:35:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:35:12 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 15:35:17 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 15:35:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 15:35:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 15:35:27 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/13 15:35:27 No eventing-con process running
2021/12/13 15:35:27 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/13 15:35:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:35:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:35:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:35:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:35:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:35:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:35:50 Waiting for dst bucket item count to get to: 0 curr count: 94610
2021/12/13 15:35:55 Waiting for dst bucket item count to get to: 0 curr count: 94610
2021/12/13 15:35:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.29s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/13 15:36:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 15:36:01 Removing node(s): [] from the cluster
2021/12/13 15:36:06 Rebalance progress: 100
2021/12/13 15:36:11 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 15:36:11 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/13 15:36:11 No eventing-con process running
2021/12/13 15:36:11 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/13 15:36:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:36:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:36:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:36:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:36:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:36:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:36:29 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:36:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 15:36:36 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 15:36:41 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 15:36:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 15:36:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 15:36:51 App: TestDuplicateWithKVRebOut got deployed
2021/12/13 15:36:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:36:51 Starting bucket ops to source bucket
2021/12/13 15:36:56 Rebalance progress: 16.66666666666667
2021/12/13 15:37:01 Rebalance progress: 16.66666666666667
2021/12/13 15:37:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:37:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:37:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:37:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:37:06 Waiting for src bucket item count to get to: 200000 curr count: 13455
2021/12/13 15:37:11 Waiting for src bucket item count to get to: 200000 curr count: 29327
2021/12/13 15:37:16 Waiting for src bucket item count to get to: 200000 curr count: 29327
2021/12/13 15:37:21 Waiting for src bucket item count to get to: 200000 curr count: 45395
2021/12/13 15:37:26 Waiting for src bucket item count to get to: 200000 curr count: 45395
2021/12/13 15:37:31 Waiting for src bucket item count to get to: 200000 curr count: 61772
2021/12/13 15:37:36 Waiting for src bucket item count to get to: 200000 curr count: 61772
2021/12/13 15:37:41 Waiting for src bucket item count to get to: 200000 curr count: 61772
2021/12/13 15:37:46 Waiting for src bucket item count to get to: 200000 curr count: 77455
2021/12/13 15:37:51 Waiting for src bucket item count to get to: 200000 curr count: 77455
2021/12/13 15:37:56 Waiting for src bucket item count to get to: 200000 curr count: 93311
2021/12/13 15:38:01 Waiting for src bucket item count to get to: 200000 curr count: 93311
2021/12/13 15:38:06 Waiting for src bucket item count to get to: 200000 curr count: 109603
2021/12/13 15:38:11 Waiting for src bucket item count to get to: 200000 curr count: 109603
2021/12/13 15:38:16 Waiting for src bucket item count to get to: 200000 curr count: 125535
2021/12/13 15:38:21 Waiting for src bucket item count to get to: 200000 curr count: 125535
2021/12/13 15:38:26 Waiting for src bucket item count to get to: 200000 curr count: 141553
2021/12/13 15:38:31 Waiting for src bucket item count to get to: 200000 curr count: 141553
2021/12/13 15:38:36 Waiting for src bucket item count to get to: 200000 curr count: 157918
2021/12/13 15:38:41 Waiting for src bucket item count to get to: 200000 curr count: 157918
2021/12/13 15:38:46 Waiting for src bucket item count to get to: 200000 curr count: 173155
2021/12/13 15:38:51 Waiting for src bucket item count to get to: 200000 curr count: 173155
2021/12/13 15:38:56 Waiting for src bucket item count to get to: 200000 curr count: 189119
2021/12/13 15:39:01 Waiting for src bucket item count to get to: 200000 curr count: 189119
2021/12/13 15:39:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:39:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:39:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:39:11 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 15:39:16 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 15:39:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 15:39:26 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 15:39:26 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/13 15:39:26 No eventing-con process running
2021/12/13 15:39:26 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/13 15:39:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:39:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:39:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:39:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:39:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.18s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/13 15:39:55 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 15:39:55 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/13 15:39:55 No eventing-con process running
2021/12/13 15:39:55 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/13 15:39:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:40:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:40:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:40:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:40:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:40:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:40:13 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:40:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 15:40:19 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 15:40:24 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 15:40:29 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 15:40:34 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 15:40:34 App: TestDuplicateWithKVSwapReb got deployed
2021/12/13 15:40:34 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 15:40:34 Starting bucket ops to source bucket
2021/12/13 15:40:35 Removing node(s): [] from the cluster
2021/12/13 15:40:40 Rebalance progress: 16.66666666666667
2021/12/13 15:40:45 Rebalance progress: 16.66666666666667
2021/12/13 15:40:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:40:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:40:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:40:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:40:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 15:40:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:40:55 Rebalance progress: 16.66666666666667
2021/12/13 15:41:00 Rebalance progress: 16.66666666666667
2021/12/13 15:41:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:41:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:41:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:41:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 15:41:10 Rebalance progress: 16.66666666666667
2021/12/13 15:41:15 Rebalance progress: 16.66666666666667
2021/12/13 15:41:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:41:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:41:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:41:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:41:20 Waiting for src bucket item count to get to: 200000 curr count: 54072
2021/12/13 15:41:25 Waiting for src bucket item count to get to: 200000 curr count: 70360
2021/12/13 15:41:30 Waiting for src bucket item count to get to: 200000 curr count: 85295
2021/12/13 15:41:35 Waiting for src bucket item count to get to: 200000 curr count: 85295
2021/12/13 15:41:40 Waiting for src bucket item count to get to: 200000 curr count: 101399
2021/12/13 15:41:45 Waiting for src bucket item count to get to: 200000 curr count: 101399
2021/12/13 15:41:50 Waiting for src bucket item count to get to: 200000 curr count: 117479
2021/12/13 15:41:55 Waiting for src bucket item count to get to: 200000 curr count: 117479
2021/12/13 15:42:00 Waiting for src bucket item count to get to: 200000 curr count: 133820
2021/12/13 15:42:05 Waiting for src bucket item count to get to: 200000 curr count: 133820
2021/12/13 15:42:10 Waiting for src bucket item count to get to: 200000 curr count: 133820
2021/12/13 15:42:15 Waiting for src bucket item count to get to: 200000 curr count: 150146
2021/12/13 15:42:20 Waiting for src bucket item count to get to: 200000 curr count: 150146
2021/12/13 15:42:25 Waiting for src bucket item count to get to: 200000 curr count: 166621
2021/12/13 15:42:30 Waiting for src bucket item count to get to: 200000 curr count: 166621
2021/12/13 15:42:35 Waiting for src bucket item count to get to: 200000 curr count: 182120
2021/12/13 15:42:40 Waiting for src bucket item count to get to: 200000 curr count: 182120
2021/12/13 15:42:45 Waiting for src bucket item count to get to: 200000 curr count: 198497
2021/12/13 15:42:50 Waiting for src bucket item count to get to: 200000 curr count: 198497
2021/12/13 15:42:55 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 15:42:55 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 15:42:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:43:00 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 15:43:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 15:43:10 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 15:43:15 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 15:43:15 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/13 15:43:15 No eventing-con process running
2021/12/13 15:43:15 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/13 15:43:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:43:27 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:43:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:43:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:43:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 15:43:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (234.46s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/13 15:43:49 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 15:43:49 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 15:43:49 No eventing-con process running
2021/12/13 15:43:49 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/13 15:43:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:43:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:43:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:44:08 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:44:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/13 15:44:14 Starting bucket ops to source bucket
2021/12/13 15:44:19 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 15:44:24 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 15:44:29 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 15:44:29 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:44:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:44:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:44:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:44:34 Rebalance progress: 16.66666666666667
2021/12/13 15:44:39 Rebalance progress: 16.66666666666667
2021/12/13 15:44:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:44:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:44:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:44:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:44:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:44:50 Rebalance progress: 16.66666666666667
2021/12/13 15:44:55 Rebalance progress: 16.66666666666667
2021/12/13 15:45:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:45:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 15:45:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 15:45:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:45:10 Rebalance progress: 12.5
2021/12/13 15:45:15 Rebalance progress: 12.5
2021/12/13 15:45:20 Rebalance progress: 12.5
2021/12/13 15:45:25 Rebalance progress: 56.25
2021/12/13 15:45:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:45:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:45:35 Rebalance progress: 12.5
2021/12/13 15:45:40 Rebalance progress: 12.5
2021/12/13 15:45:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:45:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:45:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 15:45:50 Rebalance progress: 12.5
2021/12/13 15:45:55 Rebalance progress: 12.5
2021/12/13 15:46:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:46:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 15:46:05 Rebalance progress: 12.5
2021/12/13 15:46:10 Rebalance progress: 12.5
2021/12/13 15:46:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:46:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:46:20 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 15:46:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 15:46:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 15:46:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 15:46:35 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 15:46:35 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:46:35 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/13 15:46:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:46:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/13 15:46:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:46:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:46:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:46:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/13 15:46:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (200.05s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/13 15:47:09 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 15:47:09 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/13 15:47:09 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:47:09 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/13 15:47:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:47:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:47:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:47:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:47:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:47:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:47:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:47:28 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:47:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/13 15:47:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 15:47:39 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 15:47:44 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 15:47:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 15:47:49 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:47:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:47:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:47:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:47:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:47:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 15:47:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:47:55 Rebalance progress: 12.5
2021/12/13 15:48:00 Rebalance progress: 12.5
2021/12/13 15:48:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:48:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:48:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:48:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 15:48:10 Rebalance progress: 12.5
2021/12/13 15:48:15 Rebalance progress: 12.5
2021/12/13 15:48:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:48:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:48:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 15:48:30 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 15:48:35 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 15:48:40 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 15:48:40 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/13 15:48:40 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:48:41 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/13 15:48:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:48:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:48:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4634
2021/12/13 15:48:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4634
2021/12/13 15:48:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:48:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:49:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4634
2021/12/13 15:49:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.95s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/13 15:49:19 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 15:49:19 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 15:49:19 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:49:19 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/13 15:49:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:49:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:49:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:49:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:49:38 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:49:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 15:49:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 15:49:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 15:49:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 15:49:59 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 15:49:59 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:49:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:49:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:50:04 Rebalance progress: 12.5
2021/12/13 15:50:09 Rebalance progress: 12.5
2021/12/13 15:50:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:50:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:50:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:50:20 Rebalance progress: 12.5
2021/12/13 15:50:25 Rebalance progress: 12.5
2021/12/13 15:50:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:50:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 15:50:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:50:35 Rebalance progress: 12.5
2021/12/13 15:50:40 Rebalance progress: 12.5
2021/12/13 15:50:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:50:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:50:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:50:50 Rebalance progress: 12.5
2021/12/13 15:50:55 Rebalance progress: 12.5
2021/12/13 15:51:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:51:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 15:51:06 Rebalance progress: 12.5
2021/12/13 15:51:11 Rebalance progress: 12.5
2021/12/13 15:51:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:51:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 15:51:21 Rebalance progress: 12.5
2021/12/13 15:51:26 Rebalance progress: 12.5
2021/12/13 15:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:51:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:51:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 15:51:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 15:51:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 15:51:51 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 15:51:51 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 15:51:51 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:51:51 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/13 15:51:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:51:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:52:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10684
2021/12/13 15:52:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10684
2021/12/13 15:52:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:52:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:52:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (185.47s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/13 15:52:25 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 15:52:25 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/13 15:52:25 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:52:25 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/13 15:52:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:52:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:52:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:52:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:52:38 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:52:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 15:52:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 15:52:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 15:52:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 15:52:59 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 15:52:59 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:52:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:52:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:53:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:53:05 Rebalance progress: 12.5
2021/12/13 15:53:10 Rebalance progress: 12.5
2021/12/13 15:53:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:53:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:53:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 15:53:20 Rebalance progress: 12.5
2021/12/13 15:53:25 Rebalance progress: 12.5
2021/12/13 15:53:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:53:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 15:53:35 Rebalance progress: 12.5
2021/12/13 15:53:40 Rebalance progress: 12.5
2021/12/13 15:53:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:53:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:53:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:53:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 15:53:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 15:54:00 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 15:54:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 15:54:05 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/13 15:54:05 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:54:05 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/13 15:54:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:54:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:54:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6082
2021/12/13 15:54:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:54:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:54:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:54:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6082
2021/12/13 15:54:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (134.39s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/13 15:54:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:54:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:54:39 Rebalance progress: 100
2021/12/13 15:54:44 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 15:54:44 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/13 15:54:44 No eventing-con process running
2021/12/13 15:54:44 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/13 15:54:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:54:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:54:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:54:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:54:58 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:54:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 15:55:04 Starting bucket ops to source bucket
2021/12/13 15:55:09 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 15:55:14 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 15:55:19 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 15:55:19 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:55:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:55:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:55:24 Rebalance progress: 12.5
2021/12/13 15:55:29 Rebalance progress: 12.5
2021/12/13 15:55:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:55:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/13 15:55:34 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 15:55:39 Rebalance progress: 12.5
2021/12/13 15:55:44 Rebalance progress: 12.5
2021/12/13 15:55:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:55:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:55:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:55:54 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 15:56:00 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 15:56:05 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 15:56:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 15:56:10 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/13 15:56:10 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:56:10 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/13 15:56:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:56:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:56:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4546
2021/12/13 15:56:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4546
2021/12/13 15:56:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:56:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:56:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:56:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4546
2021/12/13 15:56:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.32s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/13 15:56:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:56:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:56:49 Rebalance progress: 100
2021/12/13 15:56:54 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 15:56:54 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/13 15:56:54 No eventing-con process running
2021/12/13 15:56:54 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/13 15:56:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:57:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:57:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:57:07 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:57:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 15:57:13 Starting bucket ops to source bucket
2021/12/13 15:57:18 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 15:57:23 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 15:57:28 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 15:57:28 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:57:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 15:57:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:57:34 Rebalance progress: 12.5
2021/12/13 15:57:39 Rebalance progress: 12.5
2021/12/13 15:57:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:57:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/13 15:57:44 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 15:57:49 Rebalance progress: 12.5
2021/12/13 15:57:54 Rebalance progress: 12.5
2021/12/13 15:57:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:57:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:57:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 15:57:59 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 15:58:04 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 15:58:09 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 15:58:14 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 15:58:19 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 15:58:19 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/13 15:58:19 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:58:19 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/13 15:58:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:58:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:58:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 34550
2021/12/13 15:58:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 34550
2021/12/13 15:58:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:58:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:58:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:58:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37092
2021/12/13 15:58:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.54s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/13 15:58:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 15:58:58 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/13 15:58:58 eventingSetup.go:46: No eventing-con process running
2021/12/13 15:58:58 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/13 15:59:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:59:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 15:59:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:59:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:59:16 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:59:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 15:59:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 15:59:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 15:59:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 15:59:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 15:59:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 15:59:39 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 15:59:39 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/13 15:59:44 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 15:59:49 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 15:59:54 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 15:59:54 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 15:59:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 15:59:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 15:59:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 15:59:59 Rebalance progress: 12.5
2021/12/13 16:00:04 Rebalance progress: 12.5
2021/12/13 16:00:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:00:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:00:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:00:15 Rebalance progress: 12.5
2021/12/13 16:00:20 Rebalance progress: 12.5
2021/12/13 16:00:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:00:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 16:00:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:00:30 Rebalance progress: 12.5
2021/12/13 16:00:35 Rebalance progress: 12.5
2021/12/13 16:00:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:00:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:00:45 Rebalance progress: 12.5
2021/12/13 16:00:50 Rebalance progress: 12.5
2021/12/13 16:00:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:00:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:01:00 Rebalance progress: 12.5
2021/12/13 16:01:05 Rebalance progress: 12.5
2021/12/13 16:01:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:01:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 16:01:15 Rebalance progress: 12.5
2021/12/13 16:01:20 Rebalance progress: 12.5
2021/12/13 16:01:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:01:25 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:01:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 16:01:36 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 16:01:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 16:01:46 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 16:01:46 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/13 16:01:46 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:01:46 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/13 16:01:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:01:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:01:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12153
2021/12/13 16:01:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:02:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:02:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:02:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3682
2021/12/13 16:02:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3682
2021/12/13 16:02:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:02:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:02:20 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 16:02:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 16:02:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 16:02:35 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 16:02:35 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/13 16:02:35 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:02:35 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/13 16:02:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:02:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:02:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:02:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:02:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:02:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (241.30s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/13 16:02:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 16:02:59 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/13 16:02:59 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:02:59 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/13 16:03:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:03:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:03:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:03:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:03:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:03:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:03:19 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:03:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 16:03:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:03:30 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 16:03:35 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 16:03:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 16:03:40 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/13 16:03:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 16:03:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 16:03:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 16:03:55 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:03:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:03:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:03:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:03:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 16:03:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:04:02 Rebalance progress: 12.5
2021/12/13 16:04:07 Rebalance progress: 12.5
2021/12/13 16:04:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:04:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 16:04:17 Rebalance progress: 12.5
2021/12/13 16:04:22 Rebalance progress: 12.5
2021/12/13 16:04:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:04:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:04:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:04:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:04:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 16:04:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 16:04:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 16:04:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 16:04:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/13 16:04:47 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:04:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/13 16:04:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:04:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:05:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1264
2021/12/13 16:05:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1264
2021/12/13 16:05:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:05:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:05:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:05:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6218
2021/12/13 16:05:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:05:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:05:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 16:05:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 16:05:33 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 16:05:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 16:05:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/13 16:05:38 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:05:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/13 16:05:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:05:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:05:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:05:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:05:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (182.75s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/13 16:06:02 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 16:06:02 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/13 16:06:02 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:06:02 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/13 16:06:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:06:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:06:20 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 16:06:20 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/13 16:06:20 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:06:20 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/13 16:06:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:06:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:34 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:06:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:06:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:06:35 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:06:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/13 16:06:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:06:46 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 16:06:51 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 16:06:56 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 16:06:56 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/13 16:07:01 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/13 16:07:06 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/13 16:07:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/13 16:07:11 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:07:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:07:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:07:17 Rebalance progress: 12.5
2021/12/13 16:07:22 Rebalance progress: 12.5
2021/12/13 16:07:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:07:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:07:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:07:33 Rebalance progress: 12.5
2021/12/13 16:07:38 Rebalance progress: 12.5
2021/12/13 16:07:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:07:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:07:48 Rebalance progress: 12.5
2021/12/13 16:07:53 Rebalance progress: 12.5
2021/12/13 16:07:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:07:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:07:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:08:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 16:08:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 16:08:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 16:08:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 16:08:18 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/13 16:08:18 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:08:18 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/13 16:08:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:08:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:08:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3092
2021/12/13 16:08:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3092
2021/12/13 16:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:08:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:08:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7633
2021/12/13 16:08:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:08:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:08:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 16:08:58 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 16:09:03 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 16:09:08 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 16:09:08 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/13 16:09:08 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:09:08 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/13 16:09:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:09:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:09:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:09:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:09:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:09:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.83s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/13 16:09:32 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:09:32 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 16:09:32 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:09:32 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/13 16:09:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:09:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:09:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:09:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:09:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:09:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:09:50 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:09:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/13 16:09:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:10:01 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:10:06 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:10:11 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:10:11 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:10:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:10:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:10:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:10:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:10:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:10:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:10:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:11:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:11:08 Rebalance progress: 12.5
2021/12/13 16:11:13 Rebalance progress: 12.5
2021/12/13 16:11:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:11:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:11:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:11:23 Rebalance progress: 12.5
2021/12/13 16:11:28 Rebalance progress: 12.5
2021/12/13 16:11:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:11:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:11:38 Rebalance progress: 12.5
2021/12/13 16:11:43 Rebalance progress: 12.5
2021/12/13 16:11:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:11:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:11:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:11:54 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:11:59 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:12:04 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:12:09 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:12:09 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 16:12:09 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:12:09 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/13 16:12:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:12:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:12:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5093
2021/12/13 16:12:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5093
2021/12/13 16:12:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:12:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:12:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:12:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11196
2021/12/13 16:12:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.56s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/13 16:12:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:12:47 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 16:12:47 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:12:47 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/13 16:12:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:12:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:12:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:12:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:13:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:13:06 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:13:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/13 16:13:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:13:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:13:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:13:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 16:13:27 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:13:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:13:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:13:47 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:13:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:13:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:13:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:14:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:14:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:14:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:14:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:14:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:14:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:15:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:15:20 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:15:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:15:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:15:51 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:15:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:16:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:16:06 Rebalance progress: 12.5
2021/12/13 16:16:11 Rebalance progress: 12.5
2021/12/13 16:16:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:16:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:16:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:16:21 Rebalance progress: 12.5
2021/12/13 16:16:26 Rebalance progress: 12.5
2021/12/13 16:16:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:16:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:16:36 Rebalance progress: 12.5
2021/12/13 16:16:41 Rebalance progress: 12.5
2021/12/13 16:16:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:16:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:16:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:16:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:16:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:17:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:17:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 16:17:07 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 16:17:07 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:17:07 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/13 16:17:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:17:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:17:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21438
2021/12/13 16:17:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:17:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:17:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:17:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6041
2021/12/13 16:17:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6041
2021/12/13 16:17:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (297.99s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/13 16:17:45 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/13 16:17:45 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/13 16:17:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:17:45 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/13 16:17:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:17:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:17:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:17:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:17:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:17:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:18:04 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:18:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 16:18:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:18:15 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/13 16:18:20 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/13 16:18:25 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/13 16:18:25 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:18:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:18:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:18:30 Rebalance progress: 12.5
2021/12/13 16:18:35 Rebalance progress: 12.5
2021/12/13 16:18:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:18:40 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:18:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:18:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:18:55 Rebalance progress: 12.5
2021/12/13 16:19:00 Rebalance progress: 12.5
2021/12/13 16:19:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:19:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:19:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:19:11 Rebalance progress: 12.5
2021/12/13 16:19:16 Rebalance progress: 12.5
2021/12/13 16:19:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:19:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:19:26 Rebalance progress: 12.5
2021/12/13 16:19:31 Rebalance progress: 12.5
2021/12/13 16:19:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:19:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:19:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:19:41 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/13 16:19:46 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/13 16:19:51 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/13 16:19:56 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/13 16:19:56 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/13 16:19:56 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:19:56 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/13 16:19:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:20:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:20:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/13 16:20:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:20:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:20:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:20:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.59s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/13 16:20:25 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/13 16:20:25 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/13 16:20:25 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:20:25 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/13 16:20:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:20:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:20:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:20:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:20:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:20:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:20:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:20:43 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:20:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 16:20:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:20:55 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/13 16:21:00 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/13 16:21:05 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/13 16:21:05 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:21:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/13 16:21:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:21:10 Rebalance progress: 12.5
2021/12/13 16:21:15 Rebalance progress: 12.5
2021/12/13 16:21:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:21:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/13 16:21:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:21:25 Rebalance progress: 12.5
2021/12/13 16:21:30 Rebalance progress: 12.5
2021/12/13 16:21:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:21:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:21:40 Rebalance progress: 12.5
2021/12/13 16:21:45 Rebalance progress: 12.5
2021/12/13 16:21:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:21:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:21:51 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:21:56 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/13 16:22:01 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/13 16:22:06 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/13 16:22:11 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/13 16:22:11 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/13 16:22:11 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:22:11 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/13 16:22:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:22:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:22:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/13 16:22:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/13 16:22:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:22:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:22:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:22:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6090
2021/12/13 16:22:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (144.49s)
=== RUN   TestEventingRebInWithOps
2021/12/13 16:22:49 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/13 16:22:49 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/13 16:22:49 eventingSetup.go:46: No eventing-con process running
2021/12/13 16:22:49 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/13 16:22:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:22:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:22:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:23:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:23:08 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:23:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 16:23:14 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/13 16:23:19 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/13 16:23:24 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/13 16:23:29 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/13 16:23:29 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/13 16:23:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:23:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 16:23:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 16:23:34 Rebalance progress: 12.5
2021/12/13 16:23:39 Rebalance progress: 12.5
2021/12/13 16:23:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:23:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:23:49 Rebalance progress: 12.5
2021/12/13 16:23:54 Rebalance progress: 12.5
2021/12/13 16:23:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:23:59 Waiting for src bucket item count to get to: 200000 curr count: 30712
2021/12/13 16:24:04 Waiting for src bucket item count to get to: 200000 curr count: 46705
2021/12/13 16:24:09 Waiting for src bucket item count to get to: 200000 curr count: 46705
2021/12/13 16:24:14 Waiting for src bucket item count to get to: 200000 curr count: 62487
2021/12/13 16:24:19 Waiting for src bucket item count to get to: 200000 curr count: 62487
2021/12/13 16:24:24 Waiting for src bucket item count to get to: 200000 curr count: 77381
2021/12/13 16:24:29 Waiting for src bucket item count to get to: 200000 curr count: 77381
2021/12/13 16:24:34 Waiting for src bucket item count to get to: 200000 curr count: 77381
2021/12/13 16:24:39 Waiting for src bucket item count to get to: 200000 curr count: 92880
2021/12/13 16:24:44 Waiting for src bucket item count to get to: 200000 curr count: 109054
2021/12/13 16:24:49 Waiting for src bucket item count to get to: 200000 curr count: 109054
2021/12/13 16:24:54 Waiting for src bucket item count to get to: 200000 curr count: 124948
2021/12/13 16:24:59 Waiting for src bucket item count to get to: 200000 curr count: 124948
2021/12/13 16:25:04 Waiting for src bucket item count to get to: 200000 curr count: 140881
2021/12/13 16:25:09 Waiting for src bucket item count to get to: 200000 curr count: 140881
2021/12/13 16:25:14 Waiting for src bucket item count to get to: 200000 curr count: 156893
2021/12/13 16:25:20 Waiting for src bucket item count to get to: 200000 curr count: 156893
2021/12/13 16:25:25 Waiting for src bucket item count to get to: 200000 curr count: 171893
2021/12/13 16:25:30 Waiting for src bucket item count to get to: 200000 curr count: 171893
2021/12/13 16:25:35 Waiting for src bucket item count to get to: 200000 curr count: 187389
2021/12/13 16:25:40 Waiting for src bucket item count to get to: 200000 curr count: 187389
2021/12/13 16:25:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 16:25:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 16:25:45 Starting bucket ops to source bucket
2021/12/13 16:29:34 Waiting for dst bucket item count to get to: 0 curr count: 16683
2021/12/13 16:29:39 Waiting for dst bucket item count to get to: 0 curr count: 16683
2021/12/13 16:29:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:29:39 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 16:29:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:29:44 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/13 16:29:49 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/13 16:29:54 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/13 16:29:59 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/13 16:29:59 App: TestEventingRebInWithOps got un-deployed
2021/12/13 16:29:59 No eventing-con process running
2021/12/13 16:29:59 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/13 16:30:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:30:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:30:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:30:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:30:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (453.09s)
=== RUN   TestEventingRebOutWithOps
2021/12/13 16:30:17 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:30:18 Removing node(s): [] from the cluster
2021/12/13 16:30:23 Rebalance progress: 100
2021/12/13 16:30:28 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/13 16:30:28 App: TestEventingRebOutWithOps got un-deployed
2021/12/13 16:30:28 No eventing-con process running
2021/12/13 16:30:28 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/13 16:30:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:30:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:30:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:30:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:30:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:30:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:30:46 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:30:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 16:30:52 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/13 16:30:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/13 16:31:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/13 16:31:07 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/13 16:31:07 App: TestEventingRebOutWithOps got deployed
2021/12/13 16:31:07 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:31:07 Starting bucket ops to source bucket
2021/12/13 16:31:13 Rebalance progress: 12.5
2021/12/13 16:31:18 Rebalance progress: 12.5
2021/12/13 16:31:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:31:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:31:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:31:23 Waiting for src bucket item count to get to: 200000 curr count: 17905
2021/12/13 16:31:28 Waiting for src bucket item count to get to: 200000 curr count: 17905
2021/12/13 16:31:33 Waiting for src bucket item count to get to: 200000 curr count: 33900
2021/12/13 16:31:38 Waiting for src bucket item count to get to: 200000 curr count: 33900
2021/12/13 16:31:43 Waiting for src bucket item count to get to: 200000 curr count: 49806
2021/12/13 16:31:48 Waiting for src bucket item count to get to: 200000 curr count: 49806
2021/12/13 16:31:53 Waiting for src bucket item count to get to: 200000 curr count: 49806
2021/12/13 16:31:58 Waiting for src bucket item count to get to: 200000 curr count: 66085
2021/12/13 16:32:03 Waiting for src bucket item count to get to: 200000 curr count: 66085
2021/12/13 16:32:08 Waiting for src bucket item count to get to: 200000 curr count: 81214
2021/12/13 16:32:13 Waiting for src bucket item count to get to: 200000 curr count: 81214
2021/12/13 16:32:18 Waiting for src bucket item count to get to: 200000 curr count: 97454
2021/12/13 16:32:23 Waiting for src bucket item count to get to: 200000 curr count: 97454
2021/12/13 16:32:28 Waiting for src bucket item count to get to: 200000 curr count: 113571
2021/12/13 16:32:33 Waiting for src bucket item count to get to: 200000 curr count: 113571
2021/12/13 16:32:38 Waiting for src bucket item count to get to: 200000 curr count: 129843
2021/12/13 16:32:43 Waiting for src bucket item count to get to: 200000 curr count: 129843
2021/12/13 16:32:48 Waiting for src bucket item count to get to: 200000 curr count: 145995
2021/12/13 16:32:53 Waiting for src bucket item count to get to: 200000 curr count: 145995
2021/12/13 16:32:58 Waiting for src bucket item count to get to: 200000 curr count: 162411
2021/12/13 16:33:03 Waiting for src bucket item count to get to: 200000 curr count: 162411
2021/12/13 16:33:08 Waiting for src bucket item count to get to: 200000 curr count: 177400
2021/12/13 16:33:13 Waiting for src bucket item count to get to: 200000 curr count: 177400
2021/12/13 16:33:18 Waiting for src bucket item count to get to: 200000 curr count: 193703
2021/12/13 16:33:23 Waiting for src bucket item count to get to: 200000 curr count: 193703
2021/12/13 16:33:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 16:33:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 16:33:28 Starting bucket ops to source bucket
2021/12/13 16:37:17 Waiting for dst bucket item count to get to: 0 curr count: 22904
2021/12/13 16:37:22 Waiting for dst bucket item count to get to: 0 curr count: 4578
2021/12/13 16:37:27 Waiting for dst bucket item count to get to: 0 curr count: 4578
2021/12/13 16:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:37:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 16:37:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:37:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/13 16:37:37 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/13 16:37:42 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/13 16:37:47 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/13 16:37:47 App: TestEventingRebOutWithOps got un-deployed
2021/12/13 16:37:47 No eventing-con process running
2021/12/13 16:37:47 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/13 16:37:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:37:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:37:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:38:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (468.22s)
=== RUN   TestEventingSwapRebWithOps
2021/12/13 16:38:11 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/13 16:38:11 App: TestEventingSwapRebWithOps got un-deployed
2021/12/13 16:38:11 No eventing-con process running
2021/12/13 16:38:11 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/13 16:38:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:38:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:38:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:38:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:38:29 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:38:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 16:38:35 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/13 16:38:40 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/13 16:38:45 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/13 16:38:50 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/13 16:38:50 App: TestEventingSwapRebWithOps got deployed
2021/12/13 16:38:50 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 16:38:50 Starting bucket ops to source bucket
2021/12/13 16:38:51 Removing node(s): [] from the cluster
2021/12/13 16:38:56 Rebalance progress: 12.5
2021/12/13 16:39:01 Rebalance progress: 12.5
2021/12/13 16:39:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:39:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 16:39:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:39:11 Rebalance progress: 12.5
2021/12/13 16:39:16 Rebalance progress: 12.5
2021/12/13 16:39:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:39:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 16:39:26 Rebalance progress: 12.5
2021/12/13 16:39:31 Rebalance progress: 12.5
2021/12/13 16:39:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:39:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:39:36 Waiting for src bucket item count to get to: 200000 curr count: 59204
2021/12/13 16:39:41 Waiting for src bucket item count to get to: 200000 curr count: 73920
2021/12/13 16:39:46 Waiting for src bucket item count to get to: 200000 curr count: 73920
2021/12/13 16:39:51 Waiting for src bucket item count to get to: 200000 curr count: 89897
2021/12/13 16:39:56 Waiting for src bucket item count to get to: 200000 curr count: 89897
2021/12/13 16:40:01 Waiting for src bucket item count to get to: 200000 curr count: 106317
2021/12/13 16:40:06 Waiting for src bucket item count to get to: 200000 curr count: 106317
2021/12/13 16:40:11 Waiting for src bucket item count to get to: 200000 curr count: 122418
2021/12/13 16:40:16 Waiting for src bucket item count to get to: 200000 curr count: 122418
2021/12/13 16:40:21 Waiting for src bucket item count to get to: 200000 curr count: 138641
2021/12/13 16:40:26 Waiting for src bucket item count to get to: 200000 curr count: 138641
2021/12/13 16:40:31 Waiting for src bucket item count to get to: 200000 curr count: 138641
2021/12/13 16:40:36 Waiting for src bucket item count to get to: 200000 curr count: 154915
2021/12/13 16:40:41 Waiting for src bucket item count to get to: 200000 curr count: 154915
2021/12/13 16:40:46 Waiting for src bucket item count to get to: 200000 curr count: 170262
2021/12/13 16:40:51 Waiting for src bucket item count to get to: 200000 curr count: 170262
2021/12/13 16:40:56 Waiting for src bucket item count to get to: 200000 curr count: 186267
2021/12/13 16:41:01 Waiting for src bucket item count to get to: 200000 curr count: 186267
2021/12/13 16:41:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 16:41:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 16:41:06 Starting bucket ops to source bucket
2021/12/13 16:44:57 Waiting for dst bucket item count to get to: 0 curr count: 4479
2021/12/13 16:45:02 Waiting for dst bucket item count to get to: 0 curr count: 4479
2021/12/13 16:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:45:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 16:45:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:45:07 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/13 16:45:12 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/13 16:45:17 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/13 16:45:22 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/13 16:45:22 App: TestEventingSwapRebWithOps got un-deployed
2021/12/13 16:45:22 No eventing-con process running
2021/12/13 16:45:22 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/13 16:45:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:45:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:45:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:45:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:45:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (455.04s)
=== RUN   TestKVRebInWithOps
2021/12/13 16:45:46 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/13 16:45:46 App: TestKVRebInWithOps got un-deployed
2021/12/13 16:45:46 No eventing-con process running
2021/12/13 16:45:46 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/13 16:45:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:45:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:45:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:45:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:46:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:46:04 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:46:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 16:46:10 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/13 16:46:15 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/13 16:46:20 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/13 16:46:25 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/13 16:46:25 App: TestKVRebInWithOps got deployed
2021/12/13 16:46:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 16:46:25 Starting bucket ops to source bucket
2021/12/13 16:46:26 Removing node(s): [] from the cluster
2021/12/13 16:46:31 Rebalance progress: 12.5
2021/12/13 16:46:36 Rebalance progress: 12.5
2021/12/13 16:46:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:46:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:46:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:46:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:46:46 Rebalance progress: 12.5
2021/12/13 16:46:51 Rebalance progress: 12.5
2021/12/13 16:46:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:46:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:46:56 Waiting for src bucket item count to get to: 200000 curr count: 35780
2021/12/13 16:47:01 Waiting for src bucket item count to get to: 200000 curr count: 51692
2021/12/13 16:47:06 Waiting for src bucket item count to get to: 200000 curr count: 51692
2021/12/13 16:47:11 Waiting for src bucket item count to get to: 200000 curr count: 67796
2021/12/13 16:47:16 Waiting for src bucket item count to get to: 200000 curr count: 67796
2021/12/13 16:47:21 Waiting for src bucket item count to get to: 200000 curr count: 82747
2021/12/13 16:47:26 Waiting for src bucket item count to get to: 200000 curr count: 82747
2021/12/13 16:47:31 Waiting for src bucket item count to get to: 200000 curr count: 82747
2021/12/13 16:47:36 Waiting for src bucket item count to get to: 200000 curr count: 99035
2021/12/13 16:47:41 Waiting for src bucket item count to get to: 200000 curr count: 99035
2021/12/13 16:47:46 Waiting for src bucket item count to get to: 200000 curr count: 115399
2021/12/13 16:47:51 Waiting for src bucket item count to get to: 200000 curr count: 115399
2021/12/13 16:47:56 Waiting for src bucket item count to get to: 200000 curr count: 131607
2021/12/13 16:48:01 Waiting for src bucket item count to get to: 200000 curr count: 131607
2021/12/13 16:48:06 Waiting for src bucket item count to get to: 200000 curr count: 147789
2021/12/13 16:48:11 Waiting for src bucket item count to get to: 200000 curr count: 147789
2021/12/13 16:48:16 Waiting for src bucket item count to get to: 200000 curr count: 163611
2021/12/13 16:48:21 Waiting for src bucket item count to get to: 200000 curr count: 163611
2021/12/13 16:48:26 Waiting for src bucket item count to get to: 200000 curr count: 178315
2021/12/13 16:48:31 Waiting for src bucket item count to get to: 200000 curr count: 178315
2021/12/13 16:48:36 Waiting for src bucket item count to get to: 200000 curr count: 194361
2021/12/13 16:48:41 Waiting for src bucket item count to get to: 200000 curr count: 194361
2021/12/13 16:48:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 16:48:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 16:48:46 Starting bucket ops to source bucket
2021/12/13 16:52:35 Waiting for dst bucket item count to get to: 0 curr count: 20406
2021/12/13 16:52:40 Waiting for dst bucket item count to get to: 0 curr count: 2037
2021/12/13 16:52:45 Waiting for dst bucket item count to get to: 0 curr count: 2037
2021/12/13 16:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:52:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 16:52:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 16:52:51 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/13 16:52:56 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/13 16:53:01 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/13 16:53:06 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/13 16:53:06 App: TestKVRebInWithOps got un-deployed
2021/12/13 16:53:06 No eventing-con process running
2021/12/13 16:53:06 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/13 16:53:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:53:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:53:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:53:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:53:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (463.58s)
=== RUN   TestKVRebOutWithOps
2021/12/13 16:53:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 16:53:25 Removing node(s): [] from the cluster
2021/12/13 16:53:30 Rebalance progress: 100
2021/12/13 16:53:35 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/13 16:53:35 App: TestKVRebOutWithOps got un-deployed
2021/12/13 16:53:35 No eventing-con process running
2021/12/13 16:53:35 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/13 16:53:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:53:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:53:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 16:53:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:53:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:53:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:53:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 16:53:53 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 16:53:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 16:53:59 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/13 16:54:05 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/13 16:54:10 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/13 16:54:15 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/13 16:54:15 App: TestKVRebOutWithOps got deployed
2021/12/13 16:54:15 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 16:54:15 Starting bucket ops to source bucket
2021/12/13 16:54:20 Rebalance progress: 12.5
2021/12/13 16:54:25 Rebalance progress: 12.5
2021/12/13 16:54:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:54:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 16:54:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 16:54:30 Waiting for src bucket item count to get to: 200000 curr count: 6707
2021/12/13 16:54:35 Waiting for src bucket item count to get to: 200000 curr count: 22429
2021/12/13 16:54:40 Waiting for src bucket item count to get to: 200000 curr count: 22429
2021/12/13 16:54:45 Waiting for src bucket item count to get to: 200000 curr count: 38509
2021/12/13 16:54:50 Waiting for src bucket item count to get to: 200000 curr count: 54837
2021/12/13 16:54:55 Waiting for src bucket item count to get to: 200000 curr count: 54837
2021/12/13 16:55:00 Waiting for src bucket item count to get to: 200000 curr count: 54837
2021/12/13 16:55:05 Waiting for src bucket item count to get to: 200000 curr count: 71214
2021/12/13 16:55:10 Waiting for src bucket item count to get to: 200000 curr count: 71214
2021/12/13 16:55:15 Waiting for src bucket item count to get to: 200000 curr count: 86067
2021/12/13 16:55:20 Waiting for src bucket item count to get to: 200000 curr count: 86067
2021/12/13 16:55:25 Waiting for src bucket item count to get to: 200000 curr count: 101948
2021/12/13 16:55:30 Waiting for src bucket item count to get to: 200000 curr count: 101948
2021/12/13 16:55:35 Waiting for src bucket item count to get to: 200000 curr count: 117976
2021/12/13 16:55:40 Waiting for src bucket item count to get to: 200000 curr count: 117976
2021/12/13 16:55:45 Waiting for src bucket item count to get to: 200000 curr count: 134029
2021/12/13 16:55:50 Waiting for src bucket item count to get to: 200000 curr count: 134029
2021/12/13 16:55:55 Waiting for src bucket item count to get to: 200000 curr count: 149835
2021/12/13 16:56:00 Waiting for src bucket item count to get to: 200000 curr count: 149835
2021/12/13 16:56:05 Waiting for src bucket item count to get to: 200000 curr count: 165862
2021/12/13 16:56:10 Waiting for src bucket item count to get to: 200000 curr count: 165862
2021/12/13 16:56:15 Waiting for src bucket item count to get to: 200000 curr count: 180347
2021/12/13 16:56:20 Waiting for src bucket item count to get to: 200000 curr count: 180347
2021/12/13 16:56:25 Waiting for src bucket item count to get to: 200000 curr count: 196275
2021/12/13 16:56:30 Waiting for src bucket item count to get to: 200000 curr count: 196275
2021/12/13 16:56:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 16:56:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 16:56:35 Starting bucket ops to source bucket
2021/12/13 17:00:24 Waiting for dst bucket item count to get to: 0 curr count: 18067
2021/12/13 17:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:00:24 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:00:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:00:29 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/13 17:00:34 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/13 17:00:39 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/13 17:00:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/13 17:00:44 App: TestKVRebOutWithOps got un-deployed
2021/12/13 17:00:44 No eventing-con process running
2021/12/13 17:00:44 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/13 17:00:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:00:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:00:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (458.78s)
=== RUN   TestKVSwapRebWithOps
2021/12/13 17:01:08 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/13 17:01:08 App: TestKVSwapRebWithOps got un-deployed
2021/12/13 17:01:08 No eventing-con process running
2021/12/13 17:01:08 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/13 17:01:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:01:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:01:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:01:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:01:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:01:27 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:01:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/13 17:01:33 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/13 17:01:38 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/13 17:01:43 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/13 17:01:48 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/13 17:01:48 App: TestKVSwapRebWithOps got deployed
2021/12/13 17:01:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:01:48 Starting bucket ops to source bucket
2021/12/13 17:01:48 Removing node(s): [] from the cluster
2021/12/13 17:01:54 Rebalance progress: 12.5
2021/12/13 17:01:59 Rebalance progress: 12.5
2021/12/13 17:02:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:02:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 17:02:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:02:09 Rebalance progress: 12.5
2021/12/13 17:02:14 Rebalance progress: 12.5
2021/12/13 17:02:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:02:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 17:02:24 Rebalance progress: 12.5
2021/12/13 17:02:29 Rebalance progress: 12.5
2021/12/13 17:02:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:02:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:02:34 Waiting for src bucket item count to get to: 200000 curr count: 63726
2021/12/13 17:02:39 Waiting for src bucket item count to get to: 200000 curr count: 63726
2021/12/13 17:02:44 Waiting for src bucket item count to get to: 200000 curr count: 78477
2021/12/13 17:02:49 Waiting for src bucket item count to get to: 200000 curr count: 78477
2021/12/13 17:02:54 Waiting for src bucket item count to get to: 200000 curr count: 94337
2021/12/13 17:02:59 Waiting for src bucket item count to get to: 200000 curr count: 94337
2021/12/13 17:03:04 Waiting for src bucket item count to get to: 200000 curr count: 110680
2021/12/13 17:03:09 Waiting for src bucket item count to get to: 200000 curr count: 110680
2021/12/13 17:03:14 Waiting for src bucket item count to get to: 200000 curr count: 126967
2021/12/13 17:03:19 Waiting for src bucket item count to get to: 200000 curr count: 126967
2021/12/13 17:03:24 Waiting for src bucket item count to get to: 200000 curr count: 143394
2021/12/13 17:03:29 Waiting for src bucket item count to get to: 200000 curr count: 143394
2021/12/13 17:03:34 Waiting for src bucket item count to get to: 200000 curr count: 159731
2021/12/13 17:03:39 Waiting for src bucket item count to get to: 200000 curr count: 159731
2021/12/13 17:03:44 Waiting for src bucket item count to get to: 200000 curr count: 174473
2021/12/13 17:03:49 Waiting for src bucket item count to get to: 200000 curr count: 174473
2021/12/13 17:03:54 Waiting for src bucket item count to get to: 200000 curr count: 190297
2021/12/13 17:03:59 Waiting for src bucket item count to get to: 200000 curr count: 190297
2021/12/13 17:04:04 Waiting for src bucket item count to get to: 200000 curr count: 190297
2021/12/13 17:04:09 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 17:04:09 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 17:04:09 Starting bucket ops to source bucket
2021/12/13 17:07:59 Waiting for dst bucket item count to get to: 0 curr count: 9037
2021/12/13 17:08:04 Waiting for dst bucket item count to get to: 0 curr count: 9037
2021/12/13 17:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:08:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:08:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:08:09 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/13 17:08:14 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/13 17:08:19 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/13 17:08:24 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/13 17:08:24 App: TestKVSwapRebWithOps got un-deployed
2021/12/13 17:08:24 No eventing-con process running
2021/12/13 17:08:24 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/13 17:08:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:08:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:08:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:08:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:08:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:08:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (460.07s)
=== RUN   TestEventingRebInWithTimers
2021/12/13 17:08:48 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/13 17:08:48 App: TestEventingRebInWithTimers got un-deployed
2021/12/13 17:08:48 No eventing-con process running
2021/12/13 17:08:48 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/13 17:08:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:08:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:08:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:08:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:09:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:09:07 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:09:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:09:13 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/13 17:09:18 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/13 17:09:23 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/13 17:09:28 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/13 17:09:28 App: TestEventingRebInWithTimers got deployed
2021/12/13 17:09:28 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:09:28 Starting bucket ops to source bucket
2021/12/13 17:09:28 Removing node(s): [] from the cluster
2021/12/13 17:09:33 Rebalance progress: 12.5
2021/12/13 17:09:38 Rebalance progress: 12.5
2021/12/13 17:09:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:09:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:09:48 Rebalance progress: 12.5
2021/12/13 17:09:53 Rebalance progress: 12.5
2021/12/13 17:09:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:09:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:09:58 Waiting for src bucket item count to get to: 50000 curr count: 27940
2021/12/13 17:10:03 Waiting for src bucket item count to get to: 50000 curr count: 37359
2021/12/13 17:10:08 Waiting for src bucket item count to get to: 50000 curr count: 37359
2021/12/13 17:10:13 Waiting for src bucket item count to get to: 50000 curr count: 49975
2021/12/13 17:10:18 Waiting for src bucket item count to get to: 50000 curr count: 49975
2021/12/13 17:10:28 Waiting for dst bucket item count to get to: 50000 curr count: 43166
2021/12/13 17:10:33 Waiting for dst bucket item count to get to: 50000 curr count: 43166
2021/12/13 17:10:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:10:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:10:33 Starting bucket ops to source bucket
2021/12/13 17:11:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:11:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:11:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:11:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:12:03 Waiting for dst bucket item count to get to: 0 curr count: 40885
2021/12/13 17:12:08 Waiting for dst bucket item count to get to: 0 curr count: 30709
2021/12/13 17:12:13 Waiting for dst bucket item count to get to: 0 curr count: 30709
2021/12/13 17:12:18 Waiting for dst bucket item count to get to: 0 curr count: 15290
2021/12/13 17:12:23 Waiting for dst bucket item count to get to: 0 curr count: 15290
2021/12/13 17:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:12:23 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:12:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:12:28 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/13 17:12:33 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/13 17:12:38 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/13 17:12:43 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/13 17:12:43 App: TestEventingRebInWithTimers got un-deployed
2021/12/13 17:12:43 No eventing-con process running
2021/12/13 17:12:43 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/13 17:12:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:12:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:12:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:12:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:12:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (258.86s)
=== RUN   TestEventingRebOutWithTimers
2021/12/13 17:13:02 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:13:02 Removing node(s): [] from the cluster
2021/12/13 17:13:07 Rebalance progress: 100
2021/12/13 17:13:12 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/13 17:13:12 App: TestEventingRebOutWithTimers got un-deployed
2021/12/13 17:13:12 No eventing-con process running
2021/12/13 17:13:12 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/13 17:13:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:13:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:13:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:13:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:13:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:13:31 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:13:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:13:37 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/13 17:13:42 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/13 17:13:47 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/13 17:13:52 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/13 17:13:52 App: TestEventingRebOutWithTimers got deployed
2021/12/13 17:13:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:13:52 Starting bucket ops to source bucket
2021/12/13 17:13:57 Rebalance progress: 12.5
2021/12/13 17:14:02 Rebalance progress: 12.5
2021/12/13 17:14:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:14:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:14:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:14:07 Waiting for src bucket item count to get to: 50000 curr count: 9869
2021/12/13 17:14:12 Waiting for src bucket item count to get to: 50000 curr count: 23269
2021/12/13 17:14:17 Waiting for src bucket item count to get to: 50000 curr count: 23269
2021/12/13 17:14:22 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/13 17:14:27 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/13 17:14:32 Waiting for src bucket item count to get to: 50000 curr count: 34857
2021/12/13 17:14:37 Waiting for src bucket item count to get to: 50000 curr count: 44623
2021/12/13 17:14:42 Waiting for src bucket item count to get to: 50000 curr count: 44623
2021/12/13 17:14:52 Waiting for dst bucket item count to get to: 50000 curr count: 38353
2021/12/13 17:14:57 Waiting for dst bucket item count to get to: 50000 curr count: 38353
2021/12/13 17:14:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:14:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:14:57 Starting bucket ops to source bucket
2021/12/13 17:16:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:16:12 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:16:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:16:22 Waiting for dst bucket item count to get to: 0 curr count: 49372
2021/12/13 17:16:27 Waiting for dst bucket item count to get to: 0 curr count: 49372
2021/12/13 17:16:32 Waiting for dst bucket item count to get to: 0 curr count: 35702
2021/12/13 17:16:37 Waiting for dst bucket item count to get to: 0 curr count: 35702
2021/12/13 17:16:42 Waiting for dst bucket item count to get to: 0 curr count: 16344
2021/12/13 17:16:47 Waiting for dst bucket item count to get to: 0 curr count: 16344
2021/12/13 17:16:52 Waiting for dst bucket item count to get to: 0 curr count: 4014
2021/12/13 17:16:57 Waiting for dst bucket item count to get to: 0 curr count: 4014
2021/12/13 17:16:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:16:57 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:16:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:17:02 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/13 17:17:07 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/13 17:17:12 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/13 17:17:17 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/13 17:17:17 App: TestEventingRebOutWithTimers got un-deployed
2021/12/13 17:17:17 No eventing-con process running
2021/12/13 17:17:17 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/13 17:17:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:17:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:17:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:17:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:17:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (273.93s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/13 17:17:41 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/13 17:17:41 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/13 17:17:41 No eventing-con process running
2021/12/13 17:17:41 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/13 17:17:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:17:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:17:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:17:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:17:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:17:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:17:59 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:18:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:18:06 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/13 17:18:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/13 17:18:16 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/13 17:18:21 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/13 17:18:21 App: TestEventingSwapRebWithTimers got deployed
2021/12/13 17:18:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:18:21 Starting bucket ops to source bucket
2021/12/13 17:18:21 Removing node(s): [] from the cluster
2021/12/13 17:18:26 Rebalance progress: 12.5
2021/12/13 17:18:31 Rebalance progress: 12.5
2021/12/13 17:18:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:18:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:18:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:18:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 17:18:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:18:41 Rebalance progress: 12.5
2021/12/13 17:18:46 Rebalance progress: 12.5
2021/12/13 17:18:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:18:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:18:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:18:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 17:18:56 Rebalance progress: 12.5
2021/12/13 17:19:01 Rebalance progress: 12.5
2021/12/13 17:19:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:19:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:19:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:19:06 Waiting for src bucket item count to get to: 50000 curr count: 45665
2021/12/13 17:19:17 Waiting for dst bucket item count to get to: 50000 curr count: 36770
2021/12/13 17:19:22 Waiting for dst bucket item count to get to: 50000 curr count: 36770
2021/12/13 17:19:27 Waiting for dst bucket item count to get to: 50000 curr count: 49268
2021/12/13 17:19:32 Waiting for dst bucket item count to get to: 50000 curr count: 49268
2021/12/13 17:19:32 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:19:32 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:19:32 Starting bucket ops to source bucket
2021/12/13 17:20:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:20:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:20:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:20:56 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:21:01 Waiting for dst bucket item count to get to: 0 curr count: 39284
2021/12/13 17:21:06 Waiting for dst bucket item count to get to: 0 curr count: 39284
2021/12/13 17:21:11 Waiting for dst bucket item count to get to: 0 curr count: 29244
2021/12/13 17:21:16 Waiting for dst bucket item count to get to: 0 curr count: 29244
2021/12/13 17:21:21 Waiting for dst bucket item count to get to: 0 curr count: 12439
2021/12/13 17:21:26 Waiting for dst bucket item count to get to: 0 curr count: 12439
2021/12/13 17:21:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:21:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:21:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:21:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/13 17:21:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/13 17:21:42 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/13 17:21:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/13 17:21:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/13 17:21:47 No eventing-con process running
2021/12/13 17:21:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/13 17:21:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:21:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:21:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:21:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:22:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:22:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (269.39s)
=== RUN   TestKVRebInWithTimers
2021/12/13 17:22:10 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/13 17:22:10 App: TestKVRebInWithTimers got un-deployed
2021/12/13 17:22:10 No eventing-con process running
2021/12/13 17:22:10 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/13 17:22:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:22:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:22:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:22:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:22:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:22:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:22:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:22:29 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:22:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:22:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/13 17:22:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/13 17:22:45 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/13 17:22:50 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/13 17:22:50 App: TestKVRebInWithTimers got deployed
2021/12/13 17:22:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:22:50 Starting bucket ops to source bucket
2021/12/13 17:22:50 Removing node(s): [] from the cluster
2021/12/13 17:22:55 Rebalance progress: 12.5
2021/12/13 17:23:00 Rebalance progress: 12.5
2021/12/13 17:23:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:23:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:23:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:23:11 Rebalance progress: 12.5
2021/12/13 17:23:16 Rebalance progress: 12.5
2021/12/13 17:23:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:23:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:23:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:23:21 Waiting for src bucket item count to get to: 50000 curr count: 34917
2021/12/13 17:23:26 Waiting for src bucket item count to get to: 50000 curr count: 34917
2021/12/13 17:23:31 Waiting for src bucket item count to get to: 50000 curr count: 46333
2021/12/13 17:23:36 Waiting for src bucket item count to get to: 50000 curr count: 46333
2021/12/13 17:23:46 Waiting for dst bucket item count to get to: 50000 curr count: 42273
2021/12/13 17:23:51 Waiting for dst bucket item count to get to: 50000 curr count: 42273
2021/12/13 17:23:51 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:23:51 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:23:51 Starting bucket ops to source bucket
2021/12/13 17:25:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:25:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:25:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:25:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:25:21 Waiting for dst bucket item count to get to: 0 curr count: 43897
2021/12/13 17:25:26 Waiting for dst bucket item count to get to: 0 curr count: 43897
2021/12/13 17:25:31 Waiting for dst bucket item count to get to: 0 curr count: 21946
2021/12/13 17:25:36 Waiting for dst bucket item count to get to: 0 curr count: 21946
2021/12/13 17:25:41 Waiting for dst bucket item count to get to: 0 curr count: 7670
2021/12/13 17:25:46 Waiting for dst bucket item count to get to: 0 curr count: 7670
2021/12/13 17:25:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:25:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:25:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:25:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/13 17:25:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/13 17:26:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/13 17:26:06 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/13 17:26:06 App: TestKVRebInWithTimers got un-deployed
2021/12/13 17:26:06 No eventing-con process running
2021/12/13 17:26:06 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/13 17:26:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:26:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:26:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:26:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:26:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:26:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.66s)
=== RUN   TestKVRebOutWithTimers
2021/12/13 17:26:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:26:25 Removing node(s): [] from the cluster
2021/12/13 17:26:30 Rebalance progress: 100
2021/12/13 17:26:35 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/13 17:26:35 App: TestKVRebOutWithTimers got un-deployed
2021/12/13 17:26:35 No eventing-con process running
2021/12/13 17:26:35 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/13 17:26:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:26:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:26:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:26:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:26:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:26:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:26:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:26:54 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:26:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:27:00 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/13 17:27:05 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/13 17:27:10 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/13 17:27:15 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/13 17:27:15 App: TestKVRebOutWithTimers got deployed
2021/12/13 17:27:15 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:27:15 Starting bucket ops to source bucket
2021/12/13 17:27:20 Rebalance progress: 12.5
2021/12/13 17:27:25 Rebalance progress: 12.5
2021/12/13 17:27:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:27:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:27:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:27:30 Waiting for src bucket item count to get to: 50000 curr count: 18942
2021/12/13 17:27:35 Waiting for src bucket item count to get to: 50000 curr count: 18942
2021/12/13 17:27:40 Waiting for src bucket item count to get to: 50000 curr count: 29097
2021/12/13 17:27:45 Waiting for src bucket item count to get to: 50000 curr count: 29097
2021/12/13 17:27:50 Waiting for src bucket item count to get to: 50000 curr count: 29097
2021/12/13 17:27:55 Waiting for src bucket item count to get to: 50000 curr count: 41511
2021/12/13 17:28:00 Waiting for src bucket item count to get to: 50000 curr count: 41511
2021/12/13 17:28:10 Waiting for dst bucket item count to get to: 50000 curr count: 40318
2021/12/13 17:28:15 Waiting for dst bucket item count to get to: 50000 curr count: 40318
2021/12/13 17:28:20 Waiting for dst bucket item count to get to: 50000 curr count: 48371
2021/12/13 17:28:25 Waiting for dst bucket item count to get to: 50000 curr count: 48371
2021/12/13 17:28:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:28:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:28:25 Starting bucket ops to source bucket
2021/12/13 17:29:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:29:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:29:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:29:50 Waiting for dst bucket item count to get to: 0 curr count: 47985
2021/12/13 17:29:55 Waiting for dst bucket item count to get to: 0 curr count: 47985
2021/12/13 17:30:00 Waiting for dst bucket item count to get to: 0 curr count: 33713
2021/12/13 17:30:05 Waiting for dst bucket item count to get to: 0 curr count: 33713
2021/12/13 17:30:10 Waiting for dst bucket item count to get to: 0 curr count: 10195
2021/12/13 17:30:15 Waiting for dst bucket item count to get to: 0 curr count: 10195
2021/12/13 17:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:30:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:30:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:30:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/13 17:30:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/13 17:30:31 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/13 17:30:36 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/13 17:30:36 App: TestKVRebOutWithTimers got un-deployed
2021/12/13 17:30:36 No eventing-con process running
2021/12/13 17:30:36 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/13 17:30:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:30:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:30:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:30:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:30:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (269.42s)
=== RUN   TestKVSwapRebWithTimers
2021/12/13 17:30:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:30:59 App: TestKVSwapRebWithTimers got un-deployed
2021/12/13 17:30:59 No eventing-con process running
2021/12/13 17:30:59 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/13 17:31:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:31:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:31:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:31:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:31:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:31:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:31:18 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:31:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/13 17:31:24 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/13 17:31:29 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/13 17:31:34 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/13 17:31:39 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/13 17:31:39 App: TestKVSwapRebWithTimers got deployed
2021/12/13 17:31:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:31:39 Starting bucket ops to source bucket
2021/12/13 17:31:39 Removing node(s): [] from the cluster
2021/12/13 17:31:45 Rebalance progress: 12.5
2021/12/13 17:31:50 Rebalance progress: 12.5
2021/12/13 17:31:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:31:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:31:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:31:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 17:31:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:32:00 Rebalance progress: 12.5
2021/12/13 17:32:05 Rebalance progress: 12.5
2021/12/13 17:32:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:32:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 17:32:15 Rebalance progress: 12.5
2021/12/13 17:32:20 Rebalance progress: 12.5
2021/12/13 17:32:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:32:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:32:25 Waiting for src bucket item count to get to: 50000 curr count: 46074
2021/12/13 17:32:30 Waiting for src bucket item count to get to: 50000 curr count: 46074
2021/12/13 17:32:40 Waiting for dst bucket item count to get to: 50000 curr count: 43838
2021/12/13 17:32:45 Waiting for dst bucket item count to get to: 50000 curr count: 43838
2021/12/13 17:32:50 Waiting for dst bucket item count to get to: 50000 curr count: 49058
2021/12/13 17:32:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:32:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:32:50 Starting bucket ops to source bucket
2021/12/13 17:34:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:34:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:34:10 Waiting for dst bucket item count to get to: 0 curr count: 47682
2021/12/13 17:34:15 Waiting for dst bucket item count to get to: 0 curr count: 47682
2021/12/13 17:34:20 Waiting for dst bucket item count to get to: 0 curr count: 35600
2021/12/13 17:34:25 Waiting for dst bucket item count to get to: 0 curr count: 35600
2021/12/13 17:34:30 Waiting for dst bucket item count to get to: 0 curr count: 24624
2021/12/13 17:34:35 Waiting for dst bucket item count to get to: 0 curr count: 24624
2021/12/13 17:34:40 Waiting for dst bucket item count to get to: 0 curr count: 3265
2021/12/13 17:34:45 Waiting for dst bucket item count to get to: 0 curr count: 3265
2021/12/13 17:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:34:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/13 17:34:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:34:50 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:34:55 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:35:00 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:35:05 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:35:10 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:35:15 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/13 17:35:15 App: TestKVSwapRebWithTimers got un-deployed
2021/12/13 17:35:15 No eventing-con process running
2021/12/13 17:35:15 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/13 17:35:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:35:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:35:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:35:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:35:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:35:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (279.52s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/13 17:35:39 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:35:39 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/13 17:35:39 No eventing-con process running
2021/12/13 17:35:39 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/13 17:35:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:35:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:35:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:35:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:35:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:35:57 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:35:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 17:36:04 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/13 17:36:09 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/13 17:36:14 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/13 17:36:19 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/13 17:36:19 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/13 17:36:19 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:36:19 Starting bucket ops to source bucket
2021/12/13 17:36:19 Removing node(s): [] from the cluster
2021/12/13 17:36:24 Rebalance progress: 12.5
2021/12/13 17:36:29 Rebalance progress: 12.5
2021/12/13 17:36:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:36:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:36:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:36:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:36:39 Rebalance progress: 12.5
2021/12/13 17:36:44 Rebalance progress: 12.5
2021/12/13 17:36:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:36:49 Waiting for src bucket item count to get to: 50000 curr count: 26750
2021/12/13 17:36:54 Waiting for src bucket item count to get to: 50000 curr count: 38383
2021/12/13 17:36:59 Waiting for src bucket item count to get to: 50000 curr count: 38383
2021/12/13 17:37:04 Waiting for src bucket item count to get to: 50000 curr count: 49354
2021/12/13 17:37:09 Waiting for src bucket item count to get to: 50000 curr count: 49354
2021/12/13 17:37:19 Waiting for dst bucket item count to get to: 50000 curr count: 36044
2021/12/13 17:37:24 Waiting for dst bucket item count to get to: 50000 curr count: 36044
2021/12/13 17:37:29 Waiting for dst bucket item count to get to: 50000 curr count: 43192
2021/12/13 17:37:34 Waiting for dst bucket item count to get to: 50000 curr count: 43192
2021/12/13 17:37:34 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:37:34 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:37:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:37:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:37:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:37:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:37:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:38:00 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:38:05 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/13 17:38:05 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/13 17:38:05 No eventing-con process running
2021/12/13 17:38:05 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/13 17:38:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:38:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:38:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:38:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:38:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:38:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.48s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/13 17:38:33 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:38:34 Removing node(s): [] from the cluster
2021/12/13 17:38:39 Rebalance progress: 100
2021/12/13 17:38:44 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:38:44 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/13 17:38:44 No eventing-con process running
2021/12/13 17:38:44 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/13 17:38:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:38:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:38:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:38:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:39:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:39:02 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:39:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 17:39:08 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/13 17:39:13 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/13 17:39:18 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/13 17:39:23 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/13 17:39:23 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/13 17:39:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:39:23 Starting bucket ops to source bucket
2021/12/13 17:39:29 Rebalance progress: 12.5
2021/12/13 17:39:34 Rebalance progress: 12.5
2021/12/13 17:39:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:39:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:39:39 Waiting for src bucket item count to get to: 50000 curr count: 7287
2021/12/13 17:39:44 Waiting for src bucket item count to get to: 50000 curr count: 20840
2021/12/13 17:39:49 Waiting for src bucket item count to get to: 50000 curr count: 20840
2021/12/13 17:39:54 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/13 17:39:59 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/13 17:40:04 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/13 17:40:09 Waiting for src bucket item count to get to: 50000 curr count: 43925
2021/12/13 17:40:14 Waiting for src bucket item count to get to: 50000 curr count: 43925
2021/12/13 17:40:24 Waiting for dst bucket item count to get to: 50000 curr count: 32461
2021/12/13 17:40:29 Waiting for dst bucket item count to get to: 50000 curr count: 32461
2021/12/13 17:40:34 Waiting for dst bucket item count to get to: 50000 curr count: 41887
2021/12/13 17:40:39 Waiting for dst bucket item count to get to: 50000 curr count: 41887
2021/12/13 17:40:39 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:40:39 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:40:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:40:44 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:40:49 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:40:54 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:40:59 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:41:04 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:41:09 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/13 17:41:09 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/13 17:41:09 No eventing-con process running
2021/12/13 17:41:09 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/13 17:41:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:41:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:41:21 Waiting for dst bucket item count to get to: 0 curr count: 42978
2021/12/13 17:41:26 Waiting for dst bucket item count to get to: 0 curr count: 42978
2021/12/13 17:41:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:41:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:41:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:41:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:41:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.36s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/13 17:41:48 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:41:48 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/13 17:41:48 No eventing-con process running
2021/12/13 17:41:48 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/13 17:41:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:41:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:41:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:41:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:42:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:42:06 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:42:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 17:42:12 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/13 17:42:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/13 17:42:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/13 17:42:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/13 17:42:27 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/13 17:42:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:42:27 Starting bucket ops to source bucket
2021/12/13 17:42:28 Removing node(s): [] from the cluster
2021/12/13 17:42:33 Rebalance progress: 12.5
2021/12/13 17:42:38 Rebalance progress: 12.5
2021/12/13 17:42:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:42:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 17:42:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:42:49 Rebalance progress: 12.5
2021/12/13 17:42:54 Rebalance progress: 12.5
2021/12/13 17:42:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:42:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:42:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 17:43:04 Rebalance progress: 12.5
2021/12/13 17:43:09 Rebalance progress: 12.5
2021/12/13 17:43:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:43:14 Waiting for src bucket item count to get to: 50000 curr count: 47681
2021/12/13 17:43:19 Waiting for src bucket item count to get to: 50000 curr count: 47681
2021/12/13 17:43:29 Waiting for dst bucket item count to get to: 50000 curr count: 34206
2021/12/13 17:43:34 Waiting for dst bucket item count to get to: 50000 curr count: 34206
2021/12/13 17:43:39 Waiting for dst bucket item count to get to: 50000 curr count: 40268
2021/12/13 17:43:44 Waiting for dst bucket item count to get to: 50000 curr count: 40268
2021/12/13 17:43:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:43:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:43:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:43:49 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:43:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:43:59 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:44:04 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:44:09 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:44:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/13 17:44:14 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/13 17:44:14 No eventing-con process running
2021/12/13 17:44:14 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/13 17:44:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:44:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:44:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:44:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:44:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:44:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:44:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (180.39s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/13 17:44:48 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:44:48 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/13 17:44:48 No eventing-con process running
2021/12/13 17:44:48 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/13 17:44:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:44:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:44:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:45:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:45:07 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:45:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 17:45:13 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/13 17:45:18 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/13 17:45:23 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/13 17:45:28 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/13 17:45:28 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/13 17:45:28 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:45:28 Starting bucket ops to source bucket
2021/12/13 17:45:28 Removing node(s): [] from the cluster
2021/12/13 17:45:33 Rebalance progress: 12.5
2021/12/13 17:45:38 Rebalance progress: 12.5
2021/12/13 17:45:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:45:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:45:48 Rebalance progress: 12.5
2021/12/13 17:45:53 Rebalance progress: 12.5
2021/12/13 17:45:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:45:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:45:58 Waiting for src bucket item count to get to: 50000 curr count: 26641
2021/12/13 17:46:03 Waiting for src bucket item count to get to: 50000 curr count: 37660
2021/12/13 17:46:08 Waiting for src bucket item count to get to: 50000 curr count: 37660
2021/12/13 17:46:13 Waiting for src bucket item count to get to: 50000 curr count: 49151
2021/12/13 17:46:18 Waiting for src bucket item count to get to: 50000 curr count: 49151
2021/12/13 17:46:28 Waiting for dst bucket item count to get to: 50000 curr count: 37920
2021/12/13 17:46:33 Waiting for dst bucket item count to get to: 50000 curr count: 37920
2021/12/13 17:46:38 Waiting for dst bucket item count to get to: 50000 curr count: 45290
2021/12/13 17:46:43 Waiting for dst bucket item count to get to: 50000 curr count: 45290
2021/12/13 17:46:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:46:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:46:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:46:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:46:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:46:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:47:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:47:09 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:47:14 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/13 17:47:14 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/13 17:47:14 No eventing-con process running
2021/12/13 17:47:14 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/13 17:47:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:47:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:47:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:47:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:47:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:47:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.29s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/13 17:47:42 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:47:43 Removing node(s): [] from the cluster
2021/12/13 17:47:48 Rebalance progress: 100
2021/12/13 17:47:53 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:47:53 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/13 17:47:53 No eventing-con process running
2021/12/13 17:47:53 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/13 17:47:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:48:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:48:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:48:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:48:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:48:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:48:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:48:12 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:48:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 17:48:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/13 17:48:23 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/13 17:48:28 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/13 17:48:33 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/13 17:48:33 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/13 17:48:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:48:33 Starting bucket ops to source bucket
2021/12/13 17:48:38 Rebalance progress: 12.5
2021/12/13 17:48:43 Rebalance progress: 12.5
2021/12/13 17:48:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:48:48 Waiting for src bucket item count to get to: 50000 curr count: 8017
2021/12/13 17:48:53 Waiting for src bucket item count to get to: 50000 curr count: 21947
2021/12/13 17:48:58 Waiting for src bucket item count to get to: 50000 curr count: 21947
2021/12/13 17:49:03 Waiting for src bucket item count to get to: 50000 curr count: 35773
2021/12/13 17:49:08 Waiting for src bucket item count to get to: 50000 curr count: 35773
2021/12/13 17:49:13 Waiting for src bucket item count to get to: 50000 curr count: 45621
2021/12/13 17:49:18 Waiting for src bucket item count to get to: 50000 curr count: 45621
2021/12/13 17:49:23 Waiting for src bucket item count to get to: 50000 curr count: 45621
2021/12/13 17:49:33 Waiting for dst bucket item count to get to: 50000 curr count: 29003
2021/12/13 17:49:38 Waiting for dst bucket item count to get to: 50000 curr count: 29003
2021/12/13 17:49:43 Waiting for dst bucket item count to get to: 50000 curr count: 41937
2021/12/13 17:49:48 Waiting for dst bucket item count to get to: 50000 curr count: 41937
2021/12/13 17:49:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:49:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:49:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:49:53 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:49:58 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:50:03 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:50:08 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:50:13 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:50:18 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/13 17:50:18 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/13 17:50:18 No eventing-con process running
2021/12/13 17:50:18 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/13 17:50:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:50:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:50:30 Waiting for dst bucket item count to get to: 0 curr count: 8688
2021/12/13 17:50:35 Waiting for dst bucket item count to get to: 0 curr count: 8688
2021/12/13 17:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:50:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:50:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:50:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.62s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/13 17:50:57 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:50:57 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/13 17:50:57 No eventing-con process running
2021/12/13 17:50:57 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/13 17:50:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:51:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:51:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:51:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:51:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:51:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:51:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:51:16 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:51:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 17:51:22 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/13 17:51:27 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/13 17:51:32 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/13 17:51:37 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/13 17:51:37 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/13 17:51:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 17:51:37 Starting bucket ops to source bucket
2021/12/13 17:51:37 Removing node(s): [] from the cluster
2021/12/13 17:51:42 Rebalance progress: 12.5
2021/12/13 17:51:47 Rebalance progress: 12.5
2021/12/13 17:51:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:51:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:51:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 17:51:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:51:57 Rebalance progress: 12.5
2021/12/13 17:52:02 Rebalance progress: 12.5
2021/12/13 17:52:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:52:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:52:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 17:52:13 Rebalance progress: 12.5
2021/12/13 17:52:18 Rebalance progress: 12.5
2021/12/13 17:52:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:52:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:52:23 Waiting for src bucket item count to get to: 50000 curr count: 48108
2021/12/13 17:52:28 Waiting for src bucket item count to get to: 50000 curr count: 48108
2021/12/13 17:52:38 Waiting for dst bucket item count to get to: 50000 curr count: 39345
2021/12/13 17:52:43 Waiting for dst bucket item count to get to: 50000 curr count: 39345
2021/12/13 17:52:48 Waiting for dst bucket item count to get to: 50000 curr count: 47617
2021/12/13 17:52:53 Waiting for dst bucket item count to get to: 50000 curr count: 47617
2021/12/13 17:52:53 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:52:53 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:52:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:52:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:13 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:18 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/13 17:53:23 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/13 17:53:23 No eventing-con process running
2021/12/13 17:53:23 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/13 17:53:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:53:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:53:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:53:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:53:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:53:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.57s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/13 17:53:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:53:57 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/13 17:53:57 No eventing-con process running
2021/12/13 17:53:57 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/13 17:53:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:54:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:54:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:54:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:54:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:54:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:54:15 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:54:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 17:54:21 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/13 17:54:26 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/13 17:54:31 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/13 17:54:36 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/13 17:54:36 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/13 17:54:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:54:36 Starting bucket ops to source bucket
2021/12/13 17:54:37 Removing node(s): [] from the cluster
2021/12/13 17:54:42 Rebalance progress: 12.5
2021/12/13 17:54:47 Rebalance progress: 12.5
2021/12/13 17:54:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:54:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:54:57 Rebalance progress: 12.5
2021/12/13 17:55:02 Rebalance progress: 12.5
2021/12/13 17:55:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:55:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:55:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:55:07 Waiting for src bucket item count to get to: 50000 curr count: 27954
2021/12/13 17:55:12 Waiting for src bucket item count to get to: 50000 curr count: 38679
2021/12/13 17:55:17 Waiting for src bucket item count to get to: 50000 curr count: 38679
2021/12/13 17:55:27 Waiting for dst bucket item count to get to: 50000 curr count: 31604
2021/12/13 17:55:32 Waiting for dst bucket item count to get to: 50000 curr count: 31604
2021/12/13 17:55:37 Waiting for dst bucket item count to get to: 50000 curr count: 43118
2021/12/13 17:55:42 Waiting for dst bucket item count to get to: 50000 curr count: 43118
2021/12/13 17:55:42 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:55:42 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:55:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:55:47 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:55:52 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:55:57 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:56:02 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:56:07 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:56:12 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/13 17:56:12 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/13 17:56:12 No eventing-con process running
2021/12/13 17:56:12 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/13 17:56:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:56:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:56:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:56:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:56:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:56:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:56:41 Waiting for dst bucket item count to get to: 0 curr count: 43542
2021/12/13 17:56:46 Waiting for dst bucket item count to get to: 0 curr count: 43542
2021/12/13 17:56:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.40s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/13 17:56:51 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 17:56:51 Removing node(s): [] from the cluster
2021/12/13 17:56:56 Rebalance progress: 100
2021/12/13 17:57:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:57:01 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/13 17:57:01 No eventing-con process running
2021/12/13 17:57:01 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/13 17:57:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:57:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:57:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:57:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:57:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:57:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:57:20 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 17:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 17:57:26 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/13 17:57:31 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/13 17:57:36 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/13 17:57:41 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/13 17:57:41 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/13 17:57:41 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 17:57:41 Starting bucket ops to source bucket
2021/12/13 17:57:46 Rebalance progress: 12.5
2021/12/13 17:57:51 Rebalance progress: 12.5
2021/12/13 17:57:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 17:57:56 Waiting for src bucket item count to get to: 50000 curr count: 11273
2021/12/13 17:58:01 Waiting for src bucket item count to get to: 50000 curr count: 23276
2021/12/13 17:58:06 Waiting for src bucket item count to get to: 50000 curr count: 23276
2021/12/13 17:58:11 Waiting for src bucket item count to get to: 50000 curr count: 34180
2021/12/13 17:58:16 Waiting for src bucket item count to get to: 50000 curr count: 34180
2021/12/13 17:58:21 Waiting for src bucket item count to get to: 50000 curr count: 45439
2021/12/13 17:58:26 Waiting for src bucket item count to get to: 50000 curr count: 45439
2021/12/13 17:58:31 Waiting for src bucket item count to get to: 50000 curr count: 45439
2021/12/13 17:58:41 Waiting for dst bucket item count to get to: 50000 curr count: 41806
2021/12/13 17:58:46 Waiting for dst bucket item count to get to: 50000 curr count: 41806
2021/12/13 17:58:46 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 17:58:46 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 17:58:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 17:58:51 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:58:56 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:59:01 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:59:07 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:59:12 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:59:17 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/13 17:59:17 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/13 17:59:17 No eventing-con process running
2021/12/13 17:59:17 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/13 17:59:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:59:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:59:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 17:59:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 17:59:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 17:59:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 17:59:40 Waiting for dst bucket item count to get to: 0 curr count: 14525
2021/12/13 17:59:45 Waiting for dst bucket item count to get to: 0 curr count: 14525
2021/12/13 17:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.28s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/13 17:59:55 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 17:59:55 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/13 17:59:55 No eventing-con process running
2021/12/13 17:59:55 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/13 17:59:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:00:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:00:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:00:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:00:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:00:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:00:14 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:00:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 18:00:20 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/13 18:00:25 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/13 18:00:30 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/13 18:00:35 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/13 18:00:35 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/13 18:00:35 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 18:00:35 Starting bucket ops to source bucket
2021/12/13 18:00:36 Removing node(s): [] from the cluster
2021/12/13 18:00:41 Rebalance progress: 12.5
2021/12/13 18:00:46 Rebalance progress: 12.5
2021/12/13 18:00:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:00:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:00:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 18:00:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:00:56 Rebalance progress: 12.5
2021/12/13 18:01:01 Rebalance progress: 12.5
2021/12/13 18:01:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:01:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:01:11 Rebalance progress: 12.5
2021/12/13 18:01:16 Rebalance progress: 12.5
2021/12/13 18:01:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:01:21 Waiting for src bucket item count to get to: 50000 curr count: 49541
2021/12/13 18:01:26 Waiting for src bucket item count to get to: 50000 curr count: 49541
2021/12/13 18:01:36 Waiting for dst bucket item count to get to: 50000 curr count: 40279
2021/12/13 18:01:41 Waiting for dst bucket item count to get to: 50000 curr count: 40279
2021/12/13 18:01:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 18:01:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 18:01:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:01:46 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:01:51 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:01:56 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:02:01 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:02:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:02:12 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/13 18:02:12 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/13 18:02:12 No eventing-con process running
2021/12/13 18:02:12 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/13 18:02:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:02:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:02:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:02:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:02:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:02:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:02:40 Waiting for dst bucket item count to get to: 0 curr count: 12372
2021/12/13 18:02:45 Waiting for dst bucket item count to get to: 0 curr count: 12372
2021/12/13 18:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.96s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/13 18:02:55 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:02:55 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/13 18:02:55 No eventing-con process running
2021/12/13 18:02:55 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/13 18:02:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:03:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:03:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:03:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:03:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:03:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:03:14 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:03:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 18:03:20 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/13 18:03:25 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/13 18:03:30 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/13 18:03:35 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/13 18:03:35 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/13 18:03:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:03:35 Starting bucket ops to source bucket
2021/12/13 18:03:35 Removing node(s): [] from the cluster
2021/12/13 18:03:40 Rebalance progress: 12.5
2021/12/13 18:03:45 Rebalance progress: 12.5
2021/12/13 18:03:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:03:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:03:56 Rebalance progress: 12.5
2021/12/13 18:04:01 Rebalance progress: 12.5
2021/12/13 18:04:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:04:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:04:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:04:06 Waiting for src bucket item count to get to: 50000 curr count: 29051
2021/12/13 18:04:11 Waiting for src bucket item count to get to: 50000 curr count: 40804
2021/12/13 18:04:16 Waiting for src bucket item count to get to: 50000 curr count: 40804
2021/12/13 18:04:26 Waiting for dst bucket item count to get to: 50000 curr count: 34321
2021/12/13 18:04:31 Waiting for dst bucket item count to get to: 50000 curr count: 34321
2021/12/13 18:04:36 Waiting for dst bucket item count to get to: 50000 curr count: 44115
2021/12/13 18:04:41 Waiting for dst bucket item count to get to: 50000 curr count: 44115
2021/12/13 18:04:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 18:04:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 18:04:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:04:46 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:04:51 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:04:56 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:05:01 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:05:06 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:05:11 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/13 18:05:11 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/13 18:05:11 No eventing-con process running
2021/12/13 18:05:11 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/13 18:05:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:05:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:05:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:05:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:05:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:05:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:05:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:05:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.55s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/13 18:05:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:05:40 Removing node(s): [] from the cluster
2021/12/13 18:05:45 Rebalance progress: 100
2021/12/13 18:05:50 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:05:50 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/13 18:05:50 No eventing-con process running
2021/12/13 18:05:50 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/13 18:05:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:05:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:05:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:05:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:06:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:06:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:06:09 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:06:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 18:06:15 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/13 18:06:20 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/13 18:06:25 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/13 18:06:30 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/13 18:06:30 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/13 18:06:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:06:30 Starting bucket ops to source bucket
2021/12/13 18:06:35 Rebalance progress: 12.5
2021/12/13 18:06:40 Rebalance progress: 12.5
2021/12/13 18:06:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:06:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:06:45 Waiting for src bucket item count to get to: 50000 curr count: 13221
2021/12/13 18:06:50 Waiting for src bucket item count to get to: 50000 curr count: 13221
2021/12/13 18:06:55 Waiting for src bucket item count to get to: 50000 curr count: 26511
2021/12/13 18:07:00 Waiting for src bucket item count to get to: 50000 curr count: 36413
2021/12/13 18:07:05 Waiting for src bucket item count to get to: 50000 curr count: 36413
2021/12/13 18:07:10 Waiting for src bucket item count to get to: 50000 curr count: 48221
2021/12/13 18:07:15 Waiting for src bucket item count to get to: 50000 curr count: 48221
2021/12/13 18:07:20 Waiting for src bucket item count to get to: 50000 curr count: 48221
2021/12/13 18:07:30 Waiting for dst bucket item count to get to: 50000 curr count: 42064
2021/12/13 18:07:35 Waiting for dst bucket item count to get to: 50000 curr count: 42064
2021/12/13 18:07:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 18:07:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 18:07:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:07:40 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:07:45 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:07:50 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:07:55 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:08:00 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:08:05 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/13 18:08:05 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/13 18:08:05 No eventing-con process running
2021/12/13 18:08:05 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/13 18:08:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:08:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:08:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:08:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:08:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:08:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (174.14s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/13 18:08:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:08:39 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/13 18:08:39 No eventing-con process running
2021/12/13 18:08:39 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/13 18:08:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:08:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:08:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:08:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:08:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:08:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:08:58 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:08:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 18:09:04 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/13 18:09:09 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/13 18:09:14 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/13 18:09:19 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/13 18:09:19 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/13 18:09:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:09:19 Starting bucket ops to source bucket
2021/12/13 18:09:19 Removing node(s): [] from the cluster
2021/12/13 18:09:24 Rebalance progress: 12.5
2021/12/13 18:09:29 Rebalance progress: 12.5
2021/12/13 18:09:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:09:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:09:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:09:40 Rebalance progress: 12.5
2021/12/13 18:09:45 Rebalance progress: 12.5
2021/12/13 18:09:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:09:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:09:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:09:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:09:55 Rebalance progress: 12.5
2021/12/13 18:10:00 Rebalance progress: 12.5
2021/12/13 18:10:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:10:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:10:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:10:05 Waiting for src bucket item count to get to: 50000 curr count: 47411
2021/12/13 18:10:10 Waiting for src bucket item count to get to: 50000 curr count: 47411
2021/12/13 18:10:20 Waiting for dst bucket item count to get to: 50000 curr count: 41679
2021/12/13 18:10:25 Waiting for dst bucket item count to get to: 50000 curr count: 41679
2021/12/13 18:10:30 Waiting for dst bucket item count to get to: 50000 curr count: 49339
2021/12/13 18:10:35 Waiting for dst bucket item count to get to: 50000 curr count: 49339
2021/12/13 18:10:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/13 18:10:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/13 18:10:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:10:40 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:10:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:10:50 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:10:55 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:11:00 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:11:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/13 18:11:06 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/13 18:11:06 No eventing-con process running
2021/12/13 18:11:06 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/13 18:11:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:11:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:11:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:11:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:11:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:11:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:11:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/13 18:11:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (180.28s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/13 18:11:39 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:11:39 App: bucket_op_on_update got un-deployed
2021/12/13 18:11:39 No eventing-con process running
2021/12/13 18:11:39 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:11:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:11:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:11:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:11:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:11:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:11:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:11:58 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:11:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:12:04 Starting bucket ops to source bucket
2021/12/13 18:12:09 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:12:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:12:19 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:12:19 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:12:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:12:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:12:24 Rebalance progress: 12.5
2021/12/13 18:12:29 Rebalance progress: 12.5
2021/12/13 18:12:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:12:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:12:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:12:40 Rebalance progress: 12.5
2021/12/13 18:12:45 Rebalance progress: 12.5
2021/12/13 18:12:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:12:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:12:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 18:12:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:12:55 Rebalance progress: 12.5
2021/12/13 18:13:00 Rebalance progress: 12.5
2021/12/13 18:13:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:13:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:13:10 Rebalance progress: 12.5
2021/12/13 18:13:15 Rebalance progress: 12.5
2021/12/13 18:13:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:13:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:13:25 Rebalance progress: 12.5
2021/12/13 18:13:30 Rebalance progress: 12.5
2021/12/13 18:13:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:13:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 18:13:40 Rebalance progress: 12.5
2021/12/13 18:13:45 Rebalance progress: 12.5
2021/12/13 18:13:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:13:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:13:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:13:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:14:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:14:21 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:14:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:14:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:14:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:14:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10622
2021/12/13 18:14:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10622
2021/12/13 18:14:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:14:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:14:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:14:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10622
2021/12/13 18:14:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.02s)
=== RUN   TestKVForceFailover
2021/12/13 18:15:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:15:04 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:15:04 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:15:04 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:15:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:15:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:15:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:15:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:15:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:15:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:15:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:15:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:15:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:15:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:15:23 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:15:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:15:24 Starting bucket ops to source bucket
2021/12/13 18:15:29 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:15:34 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:15:39 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:15:44 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:15:44 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:15:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:15:44 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/13 18:16:04 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/13 18:16:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 18:16:10 Rebalance progress: 12.5
2021/12/13 18:16:15 Rebalance progress: 12.5
2021/12/13 18:16:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:16:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:16:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:16:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/13 18:16:25 Rebalance progress: 12.5
2021/12/13 18:16:30 Rebalance progress: 12.5
2021/12/13 18:16:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:16:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:16:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:16:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:16:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:16:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:17:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:17:00 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:17:00 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:17:00 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:17:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:17:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:17:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7056
2021/12/13 18:17:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7056
2021/12/13 18:17:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:17:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:17:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:17:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7056
2021/12/13 18:17:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.74s)
=== RUN   TestKVHardFailoverRecovery
2021/12/13 18:17:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:17:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:17:39 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:17:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:17:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:17:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:17:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:17:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:17:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:17:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:17:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:17:58 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:17:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:18:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:18:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:18:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:18:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:18:19 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:18:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:18:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:18:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:18:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:18:24 Rebalance progress: 12.5
2021/12/13 18:18:29 Rebalance progress: 12.5
2021/12/13 18:18:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:18:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:18:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:18:34 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/13 18:19:34 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/13 18:19:39 Rebalance progress: 12.5
2021/12/13 18:19:44 Rebalance progress: 12.5
2021/12/13 18:19:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:19:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:19:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:19:55 Rebalance progress: 12.5
2021/12/13 18:20:00 Rebalance progress: 12.5
2021/12/13 18:20:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:20:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:20:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:20:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:20:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:20:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:20:35 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:20:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:20:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:20:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:20:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12077
2021/12/13 18:20:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:20:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:20:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:20:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12077
2021/12/13 18:20:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.69s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/13 18:21:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:21:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:21:09 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:21:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:21:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:21:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:21:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:21:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:21:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:21:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:21:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:21:27 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:21:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:21:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:21:38 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:21:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:21:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:21:48 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:21:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:21:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:21:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:21:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:21:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 18:21:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:21:54 Rebalance progress: 12.5
2021/12/13 18:21:59 Rebalance progress: 12.5
2021/12/13 18:22:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:22:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 18:22:09 Rebalance progress: 12.5
2021/12/13 18:22:14 Rebalance progress: 12.5
2021/12/13 18:22:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:22:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:22:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:22:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:22:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:22:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:22:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:22:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:22:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:22:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:22:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:22:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:22:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:22:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4605
2021/12/13 18:22:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:22:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:23:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4605
2021/12/13 18:23:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.77s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/13 18:23:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:23:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:23:18 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:23:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:23:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:23:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:23:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:23:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:23:32 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:23:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:23:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:23:43 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:23:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:23:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:23:53 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:23:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:23:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:23:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:23:59 Rebalance progress: 12.5
2021/12/13 18:24:04 Rebalance progress: 12.5
2021/12/13 18:24:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:24:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:24:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:24:14 Rebalance progress: 12.5
2021/12/13 18:24:19 Rebalance progress: 12.5
2021/12/13 18:24:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:24:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:24:29 Rebalance progress: 12.5
2021/12/13 18:24:34 Rebalance progress: 12.5
2021/12/13 18:24:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:24:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:24:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:24:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:24:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:24:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:25:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:25:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:25:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:25:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:25:10 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:25:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:25:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:25:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:25:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6132
2021/12/13 18:25:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6132
2021/12/13 18:25:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:25:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:25:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:25:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6132
2021/12/13 18:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:25:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:25:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:25:44 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:25:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:25:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:25:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:25:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:25:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:25:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:25:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.74s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/13 18:26:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:26:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:26:07 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:26:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:26:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:26:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:26:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:26:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/13 18:26:26 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:26:26 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/13 18:26:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:26:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:26:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:26:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:26:40 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:26:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/13 18:26:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:26:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:26:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:27:02 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:27:02 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/13 18:27:07 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:27:12 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:27:17 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:27:17 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:27:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:27:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:27:22 Rebalance progress: 12.5
2021/12/13 18:27:27 Rebalance progress: 12.5
2021/12/13 18:27:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:27:32 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:27:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:27:38 Rebalance progress: 12.5
2021/12/13 18:27:43 Rebalance progress: 12.5
2021/12/13 18:27:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:27:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 18:27:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:27:53 Rebalance progress: 12.5
2021/12/13 18:27:58 Rebalance progress: 12.5
2021/12/13 18:28:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:28:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:28:08 Rebalance progress: 12.5
2021/12/13 18:28:13 Rebalance progress: 12.5
2021/12/13 18:28:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:28:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:28:23 Rebalance progress: 12.5
2021/12/13 18:28:28 Rebalance progress: 12.5
2021/12/13 18:28:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:28:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 18:28:39 Rebalance progress: 12.5
2021/12/13 18:28:44 Rebalance progress: 12.5
2021/12/13 18:28:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:28:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:28:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:28:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:28:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:29:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:29:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:29:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:29:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:29:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:29:19 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:29:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:29:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:29:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:29:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11416
2021/12/13 18:29:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11416
2021/12/13 18:29:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:29:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:29:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:29:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10132
2021/12/13 18:29:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10132
2021/12/13 18:29:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:29:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:30:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:30:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:30:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:30:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:30:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:30:21 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/13 18:30:21 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:30:21 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/13 18:30:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:30:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:30:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:30:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:30:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:30:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (277.77s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/13 18:30:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:30:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:30:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:30:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:30:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:30:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:30:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:30:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:30:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:30:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:31:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:31:04 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/13 18:31:04 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:31:04 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/13 18:31:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:31:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:31:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:31:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:31:17 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:31:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:31:18 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:31:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/13 18:31:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:31:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:31:34 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:31:39 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:31:39 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/13 18:31:44 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:31:49 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:31:54 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/13 18:31:54 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:31:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:31:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:31:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:32:00 Rebalance progress: 12.5
2021/12/13 18:32:05 Rebalance progress: 12.5
2021/12/13 18:32:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:32:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:32:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:32:16 Rebalance progress: 12.5
2021/12/13 18:32:21 Rebalance progress: 12.5
2021/12/13 18:32:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:32:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:32:31 Rebalance progress: 12.5
2021/12/13 18:32:36 Rebalance progress: 12.5
2021/12/13 18:32:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:32:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:32:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:32:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:32:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:32:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:33:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:33:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:33:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:33:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:33:11 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:33:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:33:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:33:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:33:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7618
2021/12/13 18:33:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:33:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:33:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:33:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7618
2021/12/13 18:33:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:33:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:33:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:33:46 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:33:51 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:33:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:34:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/13 18:34:01 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/13 18:34:01 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:34:02 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/13 18:34:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:34:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:34:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:34:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:34:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:34:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (220.30s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/13 18:34:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:34:25 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:34:25 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:34:25 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:34:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:34:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:34:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:34:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:34:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:34:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:34:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:34:44 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:34:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:34:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:34:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:35:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:35:05 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:35:05 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:35:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:35:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:35:26 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:35:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:35:41 Rebalance progress: 12.5
2021/12/13 18:35:46 Rebalance progress: 12.5
2021/12/13 18:35:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:35:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:35:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:35:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:35:57 Rebalance progress: 12.5
2021/12/13 18:36:02 Rebalance progress: 12.5
2021/12/13 18:36:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:36:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:36:12 Rebalance progress: 12.5
2021/12/13 18:36:17 Rebalance progress: 12.5
2021/12/13 18:36:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:36:22 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:36:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:36:52 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:36:52 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:36:52 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:36:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:36:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:37:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9194
2021/12/13 18:37:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9194
2021/12/13 18:37:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:37:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:37:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:37:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8331
2021/12/13 18:37:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8331
2021/12/13 18:37:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.71s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/13 18:37:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:37:36 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:37:36 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:37:36 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/13 18:37:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:37:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:37:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:37:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:37:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:37:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:37:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:37:55 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:37:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:38:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 18:38:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:38:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:38:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:38:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:38:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:38:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:38:22 Rebalance progress: 12.5
2021/12/13 18:38:27 Rebalance progress: 12.5
2021/12/13 18:38:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:38:32 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:38:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:38:47 Rebalance progress: 12.5
2021/12/13 18:38:52 Rebalance progress: 12.5
2021/12/13 18:38:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:38:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:38:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 18:38:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 18:39:02 Rebalance progress: 12.5
2021/12/13 18:39:07 Rebalance progress: 12.5
2021/12/13 18:39:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:39:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 18:39:17 Rebalance progress: 12.5
2021/12/13 18:39:22 Rebalance progress: 12.5
2021/12/13 18:39:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:39:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:39:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:39:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:39:58 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:39:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:39:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:40:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:40:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:40:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:40:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:40:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8644
2021/12/13 18:40:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (170.28s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/13 18:40:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 18:40:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 18:40:26 Rebalance progress: 100
2021/12/13 18:40:26 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:40:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:40:56 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 18:40:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 18:41:02 Removing node(s): [] from the cluster
2021/12/13 18:41:02 Starting bucket ops to source bucket
2021/12/13 18:41:08 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/13 18:41:13 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:41:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/13 18:41:18 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:41:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:41:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 18:41:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 18:41:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/13 18:41:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/13 18:41:48 eventingSetup.go:46: No eventing-con process running
2021/12/13 18:41:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/13 18:41:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:41:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:41:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 18:41:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 18:42:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 18:42:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/13 18:42:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.46s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12115.652s

Rebalance tests took 3 hours, 21 minutes and 58 seconds.

Testing Succeeded


Version: versions-13.12.2021-10.19.cfg
Build Log: make-13.12.2021-10.19.log
Server Log: logs-13.12.2021-10.19.tar.gz

Finished