Building

Started building at 2021/12/09 17:56:11
Calculating base
Updating mirror
Basing run on 7.1.0-1879 41df7530a
Updating tree for run 09.12.2021-17.56
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at efc7250, changes since last good build: none
kv_engine is at 9a6e6db, changes since last good build: 
 9a6e6db40 MB-47604: [Nexus] Add config to disable implicit compaction
 465c69e74 [Magma] Don't read manifestUid in saveDocs
 cb93fa77f MB-49839: [Magma] Cancel getAllKeys in same way that CouchKVStore does
 fdd7c5684 Fix compiler loop warning
 76d55306a Don't take ADM state lock unnecessarily
 6119cd1ca MB-49839: Adapt to GetRange API change
ns_server is at aba6a75, changes since last good build: 
 aba6a756e MB-33628 correct error message for auto-failover maximum reached
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2dbd194, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at de9f5f4, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 04a90728] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 5f1014d9] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable b5454df5] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 091cb1b8] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 421e984d] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable e58ee4ca] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 5dae5065] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable ed7672a9] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 942b026] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/09 18:14:16
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/09 18:15:47 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/09 18:15:57 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 18:16:03 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/09 18:16:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 18:16:09 Removing node(s): [] from the cluster
2021/12/09 18:16:14 Rebalance progress: 0
2021/12/09 18:16:19 Rebalance progress: 0
2021/12/09 18:16:24 Rebalance progress: 4.611545138888888
2021/12/09 18:16:29 Rebalance progress: 9.25925925925926
2021/12/09 18:16:34 Rebalance progress: 12.73148148148148
2021/12/09 18:16:39 Rebalance progress: 17.48770254629629
2021/12/09 18:16:44 Rebalance progress: 20.83333333333333
2021/12/09 18:16:49 Rebalance progress: 25.46296296296296
2021/12/09 18:16:54 Rebalance progress: 100
2021/12/09 18:16:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:17:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:17:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:17:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:17:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:18:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:18:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:18:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:18:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:19:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:19:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:19:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:19:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:20:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:20:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:20:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:20:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:21:12 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:21:27 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:21:42 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/09 18:21:57 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/09 18:22:14 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:22:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:22:21 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 18:22:26 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 18:22:31 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 18:22:36 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/09 18:22:36 App: TestAdvancedGetOps got deployed
2021/12/09 18:22:36 Starting bucket ops to source bucket
2021/12/09 18:22:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:22:46 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 18:22:46 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 18:22:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:22:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:22:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:22:51 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 18:22:56 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 18:23:01 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 18:23:06 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/09 18:23:06 App: TestAdvancedGetOps got un-deployed
2021/12/09 18:23:06 No eventing-con process running
2021/12/09 18:23:06 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/09 18:23:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:23:18 Waiting for dst bucket item count to get to: 0 curr count: 51
2021/12/09 18:23:23 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 18:23:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:23:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:23:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (80.10s)
=== RUN   TestAdvancedInsertOps
2021/12/09 18:23:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:23:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:23:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:23:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:23:51 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:23:56 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:23:56 App: TestAdvancedInsertOps got deployed
2021/12/09 18:23:56 Starting bucket ops to source bucket
2021/12/09 18:24:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 18:24:01 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 18:24:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:24:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:24:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:24:07 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:24:12 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:24:17 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:24:22 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:24:22 App: TestAdvancedInsertOps got un-deployed
2021/12/09 18:24:22 No eventing-con process running
2021/12/09 18:24:22 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 18:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:24:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:24:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 18:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:24:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:24:50 Waiting for dst bucket item count to get to: 0 curr count: 667
2021/12/09 18:24:55 Waiting for dst bucket item count to get to: 0 curr count: 667
2021/12/09 18:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:24:55 Testing insert operation on source bucket
2021/12/09 18:24:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:25:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:25:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:25:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:25:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/09 18:25:16 App: TestAdvancedInsertOps got deployed
2021/12/09 18:25:16 Starting bucket ops to source bucket
2021/12/09 18:25:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 18:25:27 Waiting for dst bucket item count to get to: 2048 curr count: 2047
2021/12/09 18:25:32 Waiting for dst bucket item count to get to: 2048 curr count: 2047
2021/12/09 18:25:32 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 18:25:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:25:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:25:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:25:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:25:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:25:47 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:25:52 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/09 18:25:52 App: TestAdvancedInsertOps got un-deployed
2021/12/09 18:25:52 No eventing-con process running
2021/12/09 18:25:52 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/09 18:25:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:26:04 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 18:26:09 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:26:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:26:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:26:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (161.50s)
=== RUN   TestAdvancedUpsertOps
2021/12/09 18:26:16 Starting bucket ops to source bucket
2021/12/09 18:26:17 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:26:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:26:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:26:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:26:33 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:26:38 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:26:38 App: TestAdvancedUpsertOps got deployed
2021/12/09 18:26:38 Starting bucket ops to source bucket
2021/12/09 18:26:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:26:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 18:26:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 18:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:26:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:26:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:26:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:26:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:27:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:27:09 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:27:14 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:27:14 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 18:27:14 No eventing-con process running
2021/12/09 18:27:14 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 18:27:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:27:26 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:27:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:27:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:27:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:27:32 Testing upsert operation on source bucket
2021/12/09 18:27:32 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:27:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:27:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:27:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:27:49 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:27:54 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/09 18:27:54 App: TestAdvancedUpsertOps got deployed
2021/12/09 18:27:54 Starting bucket ops to source bucket
2021/12/09 18:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:27:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:27:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:27:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:28:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:28:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:28:10 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:28:15 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/09 18:28:15 App: TestAdvancedUpsertOps got un-deployed
2021/12/09 18:28:15 No eventing-con process running
2021/12/09 18:28:15 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/09 18:28:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:28:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:28:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.75s)
=== RUN   TestAdvancedReplaceOps
2021/12/09 18:28:29 Starting bucket ops to source bucket
2021/12/09 18:28:30 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:28:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:28:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:28:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:28:46 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:28:51 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:28:51 App: TestAdvancedReplaceOps got deployed
2021/12/09 18:28:51 Starting bucket ops to source bucket
2021/12/09 18:28:57 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:29:02 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/09 18:29:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/09 18:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:29:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:29:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:29:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:29:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:29:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:29:22 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:29:27 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:29:27 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 18:29:27 No eventing-con process running
2021/12/09 18:29:27 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 18:29:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:29:39 Waiting for dst bucket item count to get to: 0 curr count: 954
2021/12/09 18:29:44 Waiting for dst bucket item count to get to: 0 curr count: 442
2021/12/09 18:29:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:29:50 Testing upsert operation on source bucket
2021/12/09 18:29:50 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:29:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:29:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:30:02 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:30:07 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:30:12 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/09 18:30:12 App: TestAdvancedReplaceOps got deployed
2021/12/09 18:30:12 Starting bucket ops to source bucket
2021/12/09 18:30:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:30:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:30:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:30:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:30:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:30:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:30:28 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:30:33 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/09 18:30:33 App: TestAdvancedReplaceOps got un-deployed
2021/12/09 18:30:33 No eventing-con process running
2021/12/09 18:30:33 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/09 18:30:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:30:45 Waiting for dst bucket item count to get to: 0 curr count: 213
2021/12/09 18:30:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:30:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.70s)
=== RUN   TestAdvancedDeleteOps
2021/12/09 18:30:51 Starting bucket ops to source bucket
2021/12/09 18:30:51 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:30:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:30:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:31:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:31:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:31:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:31:13 App: TestAdvancedDeleteOps got deployed
2021/12/09 18:31:13 Starting bucket ops to source bucket
2021/12/09 18:31:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:31:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:31:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:31:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:31:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:31:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:31:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:31:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:31:38 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:31:43 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:31:43 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 18:31:43 No eventing-con process running
2021/12/09 18:31:43 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 18:31:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:31:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:31:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:31:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:32:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:32:01 Testing delete operation on source bucket
2021/12/09 18:32:01 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:32:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/09 18:32:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:32:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:32:18 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:32:23 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/09 18:32:23 App: TestAdvancedDeleteOps got deployed
2021/12/09 18:32:23 Starting bucket ops to source bucket
2021/12/09 18:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:32:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:32:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:32:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:32:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:32:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:32:38 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:32:43 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/09 18:32:43 App: TestAdvancedDeleteOps got un-deployed
2021/12/09 18:32:43 No eventing-con process running
2021/12/09 18:32:43 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/09 18:32:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:32:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:32:52 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.59s)
=== RUN   TestEnoentAdvancedGet
2021/12/09 18:32:57 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:32:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 18:33:03 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:33:08 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:33:13 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:33:18 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:33:18 App: TestEnoentAdvancedGet got deployed
2021/12/09 18:33:18 Starting bucket ops to source bucket
2021/12/09 18:33:23 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:33:28 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/09 18:33:28 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 18:33:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:33:28 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:33:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:33:33 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:33:38 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:33:43 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:33:48 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:33:48 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 18:33:48 No eventing-con process running
2021/12/09 18:33:48 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 18:33:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:34:00 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/09 18:34:05 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/09 18:34:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:34:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:34:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:34:17 Testing get enoent operation on source bucket
2021/12/09 18:34:17 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:34:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/09 18:34:23 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:34:28 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:34:33 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:34:38 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/09 18:34:38 App: TestEnoentAdvancedGet got deployed
2021/12/09 18:34:38 Starting bucket ops to source bucket
2021/12/09 18:34:43 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 18:34:48 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 18:34:48 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 18:34:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:34:48 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:34:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:34:53 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:34:58 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:35:03 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:35:08 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/09 18:35:08 App: TestEnoentAdvancedGet got un-deployed
2021/12/09 18:35:08 No eventing-con process running
2021/12/09 18:35:08 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/09 18:35:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:35:20 Waiting for dst bucket item count to get to: 0 curr count: 136
2021/12/09 18:35:25 Waiting for dst bucket item count to get to: 0 curr count: 136
2021/12/09 18:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:35:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:35:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.97s)
=== RUN   TestEnoentAdvancedReplace
2021/12/09 18:35:32 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:35:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:35:38 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:35:43 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:35:48 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:35:53 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:35:53 App: TestEnoentAdvancedReplace got deployed
2021/12/09 18:35:53 Starting bucket ops to source bucket
2021/12/09 18:35:58 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:36:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:36:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 18:36:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:36:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:36:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:36:08 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:36:13 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:36:18 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:36:23 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:36:23 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 18:36:23 No eventing-con process running
2021/12/09 18:36:23 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 18:36:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:36:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:36:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:36:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:36:47 Testing get enoent operation on source bucket
2021/12/09 18:36:47 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:36:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:36:53 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:36:58 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:37:03 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:37:08 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/09 18:37:08 App: TestEnoentAdvancedReplace got deployed
2021/12/09 18:37:08 Starting bucket ops to source bucket
2021/12/09 18:37:13 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 18:37:18 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 18:37:23 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 18:37:23 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 18:37:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:37:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:37:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:37:28 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:37:33 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:37:38 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:37:43 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/09 18:37:43 App: TestEnoentAdvancedReplace got un-deployed
2021/12/09 18:37:43 No eventing-con process running
2021/12/09 18:37:43 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/09 18:37:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:37:55 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 18:37:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:37:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:38:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (149.96s)
=== RUN   TestEnoentAdvancedDelete
2021/12/09 18:38:02 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:38:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 18:38:08 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:38:13 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:38:18 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:38:23 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:38:23 App: TestEnoentAdvancedDelete got deployed
2021/12/09 18:38:23 Starting bucket ops to source bucket
2021/12/09 18:38:28 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:38:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 18:38:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 18:38:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:38:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:38:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:38:38 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:38:43 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:38:48 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:38:53 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:38:53 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 18:38:53 No eventing-con process running
2021/12/09 18:38:53 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 18:38:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:39:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:39:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:39:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:39:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:39:17 Testing get enoent operation on source bucket
2021/12/09 18:39:17 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:39:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 18:39:23 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:39:28 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:39:33 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:39:38 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/09 18:39:38 App: TestEnoentAdvancedDelete got deployed
2021/12/09 18:39:38 Starting bucket ops to source bucket
2021/12/09 18:39:43 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/09 18:39:48 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 18:39:53 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/09 18:39:53 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/09 18:39:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:39:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:39:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:39:58 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:40:03 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:40:08 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:40:13 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/09 18:40:13 App: TestEnoentAdvancedDelete got un-deployed
2021/12/09 18:40:13 No eventing-con process running
2021/12/09 18:40:13 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/09 18:40:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:40:25 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 18:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:40:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.90s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/09 18:40:32 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:40:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 18:40:38 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:40:43 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:40:48 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:40:53 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:40:53 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 18:40:53 Starting bucket ops to source bucket
2021/12/09 18:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:40:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:40:53 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:40:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:40:58 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:41:03 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:41:08 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:41:13 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:41:13 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 18:41:13 No eventing-con process running
2021/12/09 18:41:13 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 18:41:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:41:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:41:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:41:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:41:27 Pumping document in the destination bucket
2021/12/09 18:41:27 Starting bucket ops to source bucket
2021/12/09 18:41:27 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:41:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/09 18:41:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:41:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:41:44 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:41:49 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/09 18:41:49 App: TestAdvancedInsertKeyExist got deployed
2021/12/09 18:41:49 Starting bucket ops to source bucket
2021/12/09 18:41:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:41:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:41:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:41:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:41:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:41:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:42:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:42:09 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:42:14 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:42:19 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/09 18:42:19 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/09 18:42:19 No eventing-con process running
2021/12/09 18:42:19 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/09 18:42:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:42:31 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/09 18:42:36 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/09 18:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:42:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:42:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.58s)
=== RUN   TestExipryGet
2021/12/09 18:42:42 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:42:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 18:42:49 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:42:54 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:42:59 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:43:04 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:43:04 App: TestExipryGet got deployed
2021/12/09 18:43:04 Starting bucket ops to source bucket
2021/12/09 18:43:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:43:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:43:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:43:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:43:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:43:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:43:59 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:44:04 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:44:09 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:44:14 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:44:14 App: TestExipryGet got un-deployed
2021/12/09 18:44:14 No eventing-con process running
2021/12/09 18:44:14 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 18:44:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:44:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:44:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:44:28 Pumping document in the destination bucket for expiry get
2021/12/09 18:44:28 Starting bucket ops to source bucket
2021/12/09 18:44:28 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:44:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/09 18:44:34 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:44:39 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:44:44 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:44:49 Waiting for app: TestExipryGet to get deployed
2021/12/09 18:44:49 App: TestExipryGet got deployed
2021/12/09 18:44:49 Starting bucket ops to source bucket
2021/12/09 18:45:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:45:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:45:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:45:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:45:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:45:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:45:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:45:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:46:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:46:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/09 18:46:05 App: TestExipryGet got un-deployed
2021/12/09 18:46:05 No eventing-con process running
2021/12/09 18:46:05 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/09 18:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:46:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:46:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:46:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:46:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.75s)
=== RUN   TestExipryInsert
2021/12/09 18:46:23 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:46:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 18:46:29 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:46:34 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:46:39 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:46:44 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:46:44 App: TestExipryInsert got deployed
2021/12/09 18:46:44 Starting bucket ops to source bucket
2021/12/09 18:47:30 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/09 18:47:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:47:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:47:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:47:35 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:47:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:47:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:47:45 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:47:50 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:47:55 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:47:55 App: TestExipryInsert got un-deployed
2021/12/09 18:47:55 No eventing-con process running
2021/12/09 18:47:55 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 18:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:48:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:48:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:48:09 Testing on the destination bucket for expiry insert
2021/12/09 18:48:09 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:48:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/09 18:48:15 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:48:20 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:48:25 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:48:30 Waiting for app: TestExipryInsert to get deployed
2021/12/09 18:48:30 App: TestExipryInsert got deployed
2021/12/09 18:48:30 Starting bucket ops to source bucket
2021/12/09 18:49:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:49:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:49:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:49:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:49:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:49:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:49:25 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:49:30 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:49:35 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:49:40 Waiting for app: TestExipryInsert to get un-deployed
2021/12/09 18:49:40 App: TestExipryInsert got un-deployed
2021/12/09 18:49:40 No eventing-con process running
2021/12/09 18:49:40 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/09 18:49:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:49:52 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:49:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:49:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:49:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:50:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.62s)
=== RUN   TestExipryReplace
2021/12/09 18:50:04 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:50:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:50:10 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:50:15 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:50:20 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:50:25 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:50:25 App: TestExipryReplace got deployed
2021/12/09 18:50:25 Starting bucket ops to source bucket
2021/12/09 18:51:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:51:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:51:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:51:10 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:51:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:51:15 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:51:20 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:51:25 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:51:30 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:51:30 App: TestExipryReplace got un-deployed
2021/12/09 18:51:30 No eventing-con process running
2021/12/09 18:51:30 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 18:51:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:51:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:51:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:51:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:51:44 Testing on the destination bucket for expiry insert
2021/12/09 18:51:44 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:51:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:51:50 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:51:56 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:52:01 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:52:06 Waiting for app: TestExipryReplace to get deployed
2021/12/09 18:52:06 App: TestExipryReplace got deployed
2021/12/09 18:52:06 Starting bucket ops to source bucket
2021/12/09 18:52:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:52:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:52:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:52:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:52:56 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:53:01 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:53:06 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:53:11 Waiting for app: TestExipryReplace to get un-deployed
2021/12/09 18:53:11 App: TestExipryReplace got un-deployed
2021/12/09 18:53:11 No eventing-con process running
2021/12/09 18:53:11 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/09 18:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:53:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:53:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:53:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:53:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.63s)
=== RUN   TestExipryUpsert
2021/12/09 18:53:34 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:53:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 18:53:41 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:53:46 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:53:51 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:53:56 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:53:56 App: TestExipryUpsert got deployed
2021/12/09 18:53:56 Starting bucket ops to source bucket
2021/12/09 18:54:41 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:54:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:54:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:54:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:54:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:54:46 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:54:51 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:54:56 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:55:01 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:55:01 App: TestExipryUpsert got un-deployed
2021/12/09 18:55:01 No eventing-con process running
2021/12/09 18:55:01 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 18:55:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:55:15 Testing on the destination bucket for expiry insert
2021/12/09 18:55:15 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:55:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/09 18:55:21 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:55:26 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:55:31 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:55:36 Waiting for app: TestExipryUpsert to get deployed
2021/12/09 18:55:36 App: TestExipryUpsert got deployed
2021/12/09 18:55:36 Starting bucket ops to source bucket
2021/12/09 18:56:21 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:56:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:56:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:56:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:56:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:56:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:56:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:56:36 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:56:41 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/09 18:56:41 App: TestExipryUpsert got un-deployed
2021/12/09 18:56:41 No eventing-con process running
2021/12/09 18:56:41 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/09 18:56:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:56:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:56:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.55s)
=== RUN   TestCasReplace
2021/12/09 18:57:05 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:57:11 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:57:16 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:57:21 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:57:26 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:57:26 App: TestCasReplace got deployed
2021/12/09 18:57:26 Starting bucket ops to source bucket
2021/12/09 18:57:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:57:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:57:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:57:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:57:31 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:57:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:57:41 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:57:46 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:57:46 App: TestCasReplace got un-deployed
2021/12/09 18:57:46 No eventing-con process running
2021/12/09 18:57:46 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 18:57:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:57:58 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/09 18:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:58:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:58:05 Testing on the destination bucket for upsert with cas
2021/12/09 18:58:05 Starting bucket ops to source bucket
2021/12/09 18:58:05 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:58:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/09 18:58:11 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:58:16 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:58:21 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:58:26 Waiting for app: TestCasReplace to get deployed
2021/12/09 18:58:26 App: TestCasReplace got deployed
2021/12/09 18:58:26 Starting bucket ops to source bucket
2021/12/09 18:58:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:58:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:58:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:58:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:58:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:58:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:58:41 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:58:46 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:58:51 Waiting for app: TestCasReplace to get un-deployed
2021/12/09 18:58:51 App: TestCasReplace got un-deployed
2021/12/09 18:58:51 No eventing-con process running
2021/12/09 18:58:51 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/09 18:58:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:59:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 18:59:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 18:59:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:59:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 18:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.02s)
=== RUN   TestCasDelete
2021/12/09 18:59:15 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 18:59:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 18:59:21 Waiting for app: TestCasDelete to get deployed
2021/12/09 18:59:26 Waiting for app: TestCasDelete to get deployed
2021/12/09 18:59:31 Waiting for app: TestCasDelete to get deployed
2021/12/09 18:59:36 Waiting for app: TestCasDelete to get deployed
2021/12/09 18:59:36 App: TestCasDelete got deployed
2021/12/09 18:59:36 Starting bucket ops to source bucket
2021/12/09 18:59:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 18:59:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 18:59:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 18:59:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 18:59:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 18:59:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 18:59:52 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 18:59:57 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 18:59:57 App: TestCasDelete got un-deployed
2021/12/09 18:59:57 No eventing-con process running
2021/12/09 18:59:57 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 18:59:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:00:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:00:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:00:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:00:11 Testing on the destination bucket for delete with cas
2021/12/09 19:00:11 Starting bucket ops to source bucket
2021/12/09 19:00:11 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:00:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 19:00:17 Waiting for app: TestCasDelete to get deployed
2021/12/09 19:00:22 Waiting for app: TestCasDelete to get deployed
2021/12/09 19:00:27 Waiting for app: TestCasDelete to get deployed
2021/12/09 19:00:32 Waiting for app: TestCasDelete to get deployed
2021/12/09 19:00:32 App: TestCasDelete got deployed
2021/12/09 19:00:32 Starting bucket ops to source bucket
2021/12/09 19:00:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:00:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:00:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:00:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:00:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 19:00:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:00:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 19:00:52 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 19:00:57 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 19:01:02 Waiting for app: TestCasDelete to get un-deployed
2021/12/09 19:01:02 App: TestCasDelete got un-deployed
2021/12/09 19:01:02 No eventing-con process running
2021/12/09 19:01:02 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/09 19:01:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:01:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:01:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:01:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:01:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.77s)
=== RUN   TestCountersIncrement
2021/12/09 19:01:26 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 19:01:31 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 19:01:32 Removing node(s): [] from the cluster
2021/12/09 19:01:37 Rebalance progress: 100
2021/12/09 19:01:37 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:01:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 19:01:43 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:01:48 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:01:53 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:01:58 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:01:58 App: TestCountersIncrement got deployed
2021/12/09 19:01:58 Starting bucket ops to source bucket
2021/12/09 19:02:04 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 19:02:09 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 19:02:14 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/09 19:02:14 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 19:02:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:02:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:02:19 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:02:24 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:02:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:02:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:02:34 App: TestCountersIncrement got un-deployed
2021/12/09 19:02:34 No eventing-con process running
2021/12/09 19:02:34 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 19:02:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:02:46 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/09 19:02:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:02:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:02:58 Waiting for dst bucket item count to get to: 0 curr count: 645
2021/12/09 19:03:03 Waiting for dst bucket item count to get to: 0 curr count: 133
2021/12/09 19:03:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:03:03 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:03:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/09 19:03:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:03:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:03:19 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:03:24 Waiting for app: TestCountersIncrement to get deployed
2021/12/09 19:03:24 App: TestCountersIncrement got deployed
2021/12/09 19:03:24 Starting bucket ops to source bucket
2021/12/09 19:03:30 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 19:03:35 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/09 19:03:35 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 19:03:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:03:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:03:40 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:03:45 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:03:50 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:03:55 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/09 19:03:55 App: TestCountersIncrement got un-deployed
2021/12/09 19:03:55 No eventing-con process running
2021/12/09 19:03:56 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/09 19:03:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:04:07 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 19:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:04:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:04:19 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 19:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:04:19 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 19:04:24 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.35s)
=== RUN   TestCountersDecrement
2021/12/09 19:04:24 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 19:04:24 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 19:04:25 Removing node(s): [] from the cluster
2021/12/09 19:04:30 Rebalance progress: 100
2021/12/09 19:04:30 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:04:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 19:04:36 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:04:41 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:04:46 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:04:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:04:51 App: TestCountersDecrement got deployed
2021/12/09 19:04:51 Starting bucket ops to source bucket
2021/12/09 19:04:57 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 19:05:02 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/09 19:05:02 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/09 19:05:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:05:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:05:07 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:05:12 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:05:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:05:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:05:22 App: TestCountersDecrement got un-deployed
2021/12/09 19:05:22 No eventing-con process running
2021/12/09 19:05:22 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 19:05:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:05:34 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/09 19:05:39 Waiting for dst bucket item count to get to: 0 curr count: 513
2021/12/09 19:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:05:51 Waiting for dst bucket item count to get to: 0 curr count: 978
2021/12/09 19:05:56 Waiting for dst bucket item count to get to: 0 curr count: 978
2021/12/09 19:05:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:05:56 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:05:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/09 19:06:02 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:06:07 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:06:12 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:06:17 Waiting for app: TestCountersDecrement to get deployed
2021/12/09 19:06:17 App: TestCountersDecrement got deployed
2021/12/09 19:06:17 Starting bucket ops to source bucket
2021/12/09 19:06:23 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/09 19:06:28 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/09 19:06:33 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/09 19:06:33 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/09 19:06:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:06:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:06:38 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:06:43 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:06:48 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:06:53 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/09 19:06:53 App: TestCountersDecrement got un-deployed
2021/12/09 19:06:53 No eventing-con process running
2021/12/09 19:06:54 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/09 19:06:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:07:05 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/09 19:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:07:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:07:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 19:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:07:17 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/09 19:07:22 Rebalance progress: 100
--- PASS: TestCountersDecrement (178.01s)
=== RUN   TestCacheGet
2021/12/09 19:07:22 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:07:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/09 19:07:28 Waiting for app: TestCacheGet to get deployed
2021/12/09 19:07:33 Waiting for app: TestCacheGet to get deployed
2021/12/09 19:07:38 Waiting for app: TestCacheGet to get deployed
2021/12/09 19:07:43 Waiting for app: TestCacheGet to get deployed
2021/12/09 19:07:43 App: TestCacheGet got deployed
2021/12/09 19:07:43 Starting bucket ops to source bucket
2021/12/09 19:07:48 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 19:07:53 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 19:07:59 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/09 19:08:04 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/09 19:08:09 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/09 19:08:14 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/09 19:08:14 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/09 19:08:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:08:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:08:19 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 19:08:24 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 19:08:29 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 19:08:34 Waiting for app: TestCacheGet to get un-deployed
2021/12/09 19:08:34 App: TestCacheGet got un-deployed
2021/12/09 19:08:34 No eventing-con process running
2021/12/09 19:08:34 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/09 19:08:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:08:46 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 19:08:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:08:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:08:57 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/09 19:08:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.28s)
=== RUN   TestCacheRYOW
2021/12/09 19:08:57 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:08:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/09 19:09:04 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 19:09:09 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 19:09:14 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 19:09:19 Waiting for app: TestCacheRYOW to get deployed
2021/12/09 19:09:19 App: TestCacheRYOW got deployed
2021/12/09 19:09:19 Starting bucket ops to source bucket
2021/12/09 19:09:24 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/09 19:09:29 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/09 19:09:29 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/09 19:09:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:09:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:09:34 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 19:09:39 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 19:09:44 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 19:09:49 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/09 19:09:49 App: TestCacheRYOW got un-deployed
2021/12/09 19:09:49 No eventing-con process running
2021/12/09 19:09:49 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/09 19:09:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:10:01 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/09 19:10:06 Waiting for dst bucket item count to get to: 0 curr count: 46
2021/12/09 19:10:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:10:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:10:17 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/09 19:10:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.17s)
=== RUN   TestCacheExpiry
2021/12/09 19:10:18 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:10:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/09 19:10:24 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 19:10:29 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 19:10:34 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 19:10:39 Waiting for app: TestCacheExpiry to get deployed
2021/12/09 19:10:39 App: TestCacheExpiry got deployed
2021/12/09 19:10:39 Starting bucket ops to source bucket
2021/12/09 19:10:44 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:10:49 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/09 19:10:54 Waiting for dst bucket item count to get to: 100 curr count: 42
2021/12/09 19:10:59 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/09 19:11:04 Waiting for dst bucket item count to get to: 100 curr count: 97
2021/12/09 19:11:04 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:11:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:11:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:11:09 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 19:11:14 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 19:11:19 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 19:11:24 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/09 19:11:24 App: TestCacheExpiry got un-deployed
2021/12/09 19:11:24 No eventing-con process running
2021/12/09 19:11:24 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/09 19:11:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:11:36 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:11:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:11:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:11:48 Waiting for dst bucket item count to get to: 0 curr count: 58
2021/12/09 19:11:53 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/09 19:11:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (95.24s)
=== RUN   TestCacheExpiryCustom
2021/12/09 19:11:53 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:11:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 19:11:59 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 19:12:04 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 19:12:09 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 19:12:14 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/09 19:12:14 App: TestCacheExpiryCustom got deployed
2021/12/09 19:12:14 Starting bucket ops to source bucket
2021/12/09 19:12:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:12:24 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/09 19:12:29 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/09 19:12:34 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/09 19:12:39 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/09 19:12:39 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:12:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:12:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:12:44 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 19:12:49 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 19:12:54 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 19:12:59 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/09 19:12:59 App: TestCacheExpiryCustom got un-deployed
2021/12/09 19:12:59 No eventing-con process running
2021/12/09 19:13:00 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/09 19:13:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:13:11 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:13:16 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:13:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:13:28 Waiting for dst bucket item count to get to: 0 curr count: 68
2021/12/09 19:13:33 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/09 19:13:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.23s)
=== RUN   TestCacheOverflowCustom
2021/12/09 19:13:33 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:13:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/09 19:13:39 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 19:13:44 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 19:13:49 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 19:13:54 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/09 19:13:54 App: TestCacheOverflowCustom got deployed
2021/12/09 19:13:54 Starting bucket ops to source bucket
2021/12/09 19:13:59 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/09 19:14:04 Waiting for dst bucket item count to get to: 35 curr count: 8
2021/12/09 19:14:09 Waiting for dst bucket item count to get to: 35 curr count: 17
2021/12/09 19:14:14 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/09 19:14:19 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/09 19:14:24 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/09 19:14:24 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/09 19:14:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:14:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:14:29 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 19:14:34 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 19:14:39 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 19:14:44 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/09 19:14:44 App: TestCacheOverflowCustom got un-deployed
2021/12/09 19:14:44 No eventing-con process running
2021/12/09 19:14:45 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/09 19:14:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:14:56 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/09 19:14:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:14:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:15:08 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/09 19:15:13 Waiting for dst bucket item count to get to: 0 curr count: 8
2021/12/09 19:15:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.16s)
=== RUN   TestStrictMode
2021/12/09 19:15:13 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:15:14 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.06s)
=== RUN   TestDataTypes
2021/12/09 19:15:14 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:15:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/09 19:15:20 Waiting for app: TestDataTypes to get deployed
2021/12/09 19:15:25 Waiting for app: TestDataTypes to get deployed
2021/12/09 19:15:30 Waiting for app: TestDataTypes to get deployed
2021/12/09 19:15:35 Waiting for app: TestDataTypes to get deployed
2021/12/09 19:15:35 App: TestDataTypes got deployed
2021/12/09 19:15:35 Starting bucket ops to source bucket
2021/12/09 19:15:40 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/09 19:15:45 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/09 19:15:45 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/09 19:15:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:15:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:15:51 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 19:15:56 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 19:16:01 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 19:16:06 Waiting for app: TestDataTypes to get un-deployed
2021/12/09 19:16:06 App: TestDataTypes got un-deployed
2021/12/09 19:16:06 No eventing-con process running
2021/12/09 19:16:06 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/09 19:16:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:16:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:16:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:16:25 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/09 19:16:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.63s)
=== RUN   TestEnoentGet
2021/12/09 19:16:25 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:16:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 19:16:31 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:16:36 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:16:41 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:16:46 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:16:46 App: TestEnoentGet got deployed
2021/12/09 19:16:46 Starting bucket ops to source bucket
2021/12/09 19:16:51 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:16:51 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:16:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:16:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:16:56 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:17:01 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:17:06 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:17:11 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:17:11 App: TestEnoentGet got un-deployed
2021/12/09 19:17:11 No eventing-con process running
2021/12/09 19:17:11 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 19:17:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:17:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:17:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:17:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:17:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:17:40 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/09 19:17:45 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/09 19:17:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:17:45 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:17:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/09 19:17:51 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:17:56 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:18:01 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:18:06 Waiting for app: TestEnoentGet to get deployed
2021/12/09 19:18:06 App: TestEnoentGet got deployed
2021/12/09 19:18:06 Starting bucket ops to source bucket
2021/12/09 19:18:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:18:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:18:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:18:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:18:16 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:18:21 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:18:26 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:18:31 Waiting for app: TestEnoentGet to get un-deployed
2021/12/09 19:18:31 App: TestEnoentGet got un-deployed
2021/12/09 19:18:31 No eventing-con process running
2021/12/09 19:18:32 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/09 19:18:33 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:18:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:18:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:18:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:18:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:19:00 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/09 19:19:05 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/09 19:19:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.18s)
=== RUN   TestEnoentDelete
2021/12/09 19:19:05 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:19:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 19:19:11 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:19:16 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:19:21 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:19:26 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:19:26 App: TestEnoentDelete got deployed
2021/12/09 19:19:26 Starting bucket ops to source bucket
2021/12/09 19:19:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:19:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:19:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:19:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:19:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:19:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:19:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:19:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:19:52 App: TestEnoentDelete got un-deployed
2021/12/09 19:19:52 No eventing-con process running
2021/12/09 19:19:52 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 19:19:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:20:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:20:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:20:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:20:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:20:20 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/09 19:20:25 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/09 19:20:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:20:25 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:20:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 19:20:31 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:20:36 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:20:41 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:20:46 Waiting for app: TestEnoentDelete to get deployed
2021/12/09 19:20:46 App: TestEnoentDelete got deployed
2021/12/09 19:20:46 Starting bucket ops to source bucket
2021/12/09 19:20:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:20:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:20:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:20:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:20:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:21:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:21:07 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:21:12 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/09 19:21:12 App: TestEnoentDelete got un-deployed
2021/12/09 19:21:12 No eventing-con process running
2021/12/09 19:21:12 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/09 19:21:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:21:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:21:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:21:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:21:40 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/09 19:21:45 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/09 19:21:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.41s)
=== RUN   TestError
2021/12/09 19:21:55 Waiting for app: TestError to get un-deployed
2021/12/09 19:21:55 App: TestError got un-deployed
2021/12/09 19:21:55 No eventing-con process running
2021/12/09 19:21:55 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/09 19:21:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:22:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:22:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:22:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:22:09 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:22:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/09 19:22:15 Waiting for app: TestError to get deployed
2021/12/09 19:22:20 Waiting for app: TestError to get deployed
2021/12/09 19:22:25 Waiting for app: TestError to get deployed
2021/12/09 19:22:30 Waiting for app: TestError to get deployed
2021/12/09 19:22:30 App: TestError got deployed
2021/12/09 19:22:30 Starting bucket ops to source bucket
2021/12/09 19:22:35 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 19:22:40 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/09 19:22:40 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/09 19:22:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:22:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:22:46 Waiting for app: TestError to get un-deployed
2021/12/09 19:22:51 Waiting for app: TestError to get un-deployed
2021/12/09 19:22:56 Waiting for app: TestError to get un-deployed
2021/12/09 19:23:01 Waiting for app: TestError to get un-deployed
2021/12/09 19:23:01 App: TestError got un-deployed
2021/12/09 19:23:01 No eventing-con process running
2021/12/09 19:23:01 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/09 19:23:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:23:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:23:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 19:23:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:23:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:23:29 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/09 19:23:34 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/09 19:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.76s)
=== RUN   TestCRLF
2021/12/09 19:23:44 Waiting for app: TestCRLF to get un-deployed
2021/12/09 19:23:44 App: TestCRLF got un-deployed
2021/12/09 19:23:44 No eventing-con process running
2021/12/09 19:23:44 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/09 19:23:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:23:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:23:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:23:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:23:58 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:23:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/09 19:24:04 Waiting for app: TestCRLF to get deployed
2021/12/09 19:24:09 Waiting for app: TestCRLF to get deployed
2021/12/09 19:24:14 Waiting for app: TestCRLF to get deployed
2021/12/09 19:24:19 Waiting for app: TestCRLF to get deployed
2021/12/09 19:24:19 App: TestCRLF got deployed
2021/12/09 19:24:19 Starting bucket ops to source bucket
2021/12/09 19:24:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 19:24:29 Waiting for dst bucket item count to get to: 100 curr count: 53
2021/12/09 19:24:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 19:24:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:24:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:24:34 Waiting for app: TestCRLF to get un-deployed
2021/12/09 19:24:39 Waiting for app: TestCRLF to get un-deployed
2021/12/09 19:24:44 Waiting for app: TestCRLF to get un-deployed
2021/12/09 19:24:49 Waiting for app: TestCRLF to get un-deployed
2021/12/09 19:24:49 App: TestCRLF got un-deployed
2021/12/09 19:24:49 No eventing-con process running
2021/12/09 19:24:49 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/09 19:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:25:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:25:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 19:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:25:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:25:18 Waiting for dst bucket item count to get to: 0 curr count: 65
2021/12/09 19:25:23 Waiting for dst bucket item count to get to: 0 curr count: 12
2021/12/09 19:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.67s)
=== RUN   TestImportExport
2021/12/09 19:25:33 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:25:33 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 19:25:33 No eventing-con process running
2021/12/09 19:25:33 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/09 19:25:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:25:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:25:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:25:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:25:51 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:25:51 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 19:25:51 No eventing-con process running
2021/12/09 19:25:51 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/09 19:25:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:26:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:26:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:26:05 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:26:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 19:26:06 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:26:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 19:26:12 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 19:26:17 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 19:26:22 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 19:26:27 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/09 19:26:27 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/09 19:26:32 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 19:26:37 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 19:26:42 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/09 19:26:42 App: TestImportExport_bucket_op_on_update got deployed
2021/12/09 19:26:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:26:47 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:26:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:26:58 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:27:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:27:03 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 19:27:03 No eventing-con process running
2021/12/09 19:27:03 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 19:27:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:27:08 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:27:13 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:27:18 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:27:23 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:27:23 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 19:27:23 No eventing-con process running
2021/12/09 19:27:23 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 19:27:40 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/09 19:27:40 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/09 19:27:40 No eventing-con process running
2021/12/09 19:27:40 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/09 19:27:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:27:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:27:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:27:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:27:59 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/09 19:27:59 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/09 19:27:59 No eventing-con process running
2021/12/09 19:27:59 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/09 19:28:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:28:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:28:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:28:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.49s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/09 19:28:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:28:22 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:28:22 No eventing-con process running
2021/12/09 19:28:22 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/09 19:28:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:28:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:28:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:28:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:28:36 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:28:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:28:37 Starting bucket ops to source bucket
2021/12/09 19:28:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:29:00 Waiting for dst bucket item count to get to: 5000 curr count: 1514
2021/12/09 19:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 1514
2021/12/09 19:29:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:29:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 19:29:10 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 19:29:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:29:10 Undeploying app: bucket_op_on_update
2021/12/09 19:29:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:29:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:29:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:29:25 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:29:30 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:29:30 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:29:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:29:30 No eventing-con process running
2021/12/09 19:29:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:29:39 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:30:14 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:30:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:30:15 Starting bucket ops to source bucket
2021/12/09 19:30:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:30:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:30:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:30:38 Waiting for dst bucket item count to get to: 5000 curr count: 2844
2021/12/09 19:30:43 Waiting for dst bucket item count to get to: 5000 curr count: 2844
2021/12/09 19:30:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:30:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 19:30:48 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 19:30:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:30:48 Undeploying app: bucket_op_on_update
2021/12/09 19:30:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:30:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:30:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:31:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:31:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:31:09 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:31:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:31:09 No eventing-con process running
2021/12/09 19:31:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:31:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:31:53 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:31:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:31:54 Starting bucket ops to source bucket
2021/12/09 19:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:32:17 Waiting for dst bucket item count to get to: 5000 curr count: 2166
2021/12/09 19:32:22 Waiting for dst bucket item count to get to: 5000 curr count: 4666
2021/12/09 19:32:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:32:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 19:32:27 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 19:32:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:32:27 Undeploying app: bucket_op_on_update
2021/12/09 19:32:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:32:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:32:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:32:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:32:47 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:32:47 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:32:47 No eventing-con process running
2021/12/09 19:32:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:32:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:33:31 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:33:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:33:32 Starting bucket ops to source bucket
2021/12/09 19:33:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:33:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:33:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:33:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:33:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:34:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 19:34:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 19:34:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:34:00 Undeploying app: bucket_op_on_update
2021/12/09 19:34:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:34:05 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:34:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:34:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:34:20 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:34:20 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:34:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:34:20 No eventing-con process running
2021/12/09 19:34:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:34:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:35:04 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:35:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:35:05 Starting bucket ops to source bucket
2021/12/09 19:35:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:35:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:35:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:35:28 Waiting for dst bucket item count to get to: 5000 curr count: 2580
2021/12/09 19:35:33 Waiting for dst bucket item count to get to: 5000 curr count: 2580
2021/12/09 19:35:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:35:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/09 19:35:38 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/09 19:35:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:35:38 Undeploying app: bucket_op_on_update
2021/12/09 19:35:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:35:43 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:35:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:35:53 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:35:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/09 19:35:58 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/09 19:35:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:35:58 No eventing-con process running
2021/12/09 19:36:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:36:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:36:42 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (509.89s)
=== RUN   TestOnUpdateN1QLOp
2021/12/09 19:36:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 19:36:52 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 19:36:52 No eventing-con process running
2021/12/09 19:36:52 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/09 19:36:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:36:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:37:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:37:06 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:37:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/09 19:37:07 Starting bucket ops to source bucket
2021/12/09 19:37:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:37:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:37:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:37:30 Waiting for dst bucket item count to get to: 5000 curr count: 375
2021/12/09 19:37:35 Waiting for dst bucket item count to get to: 5000 curr count: 375
2021/12/09 19:37:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:37:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:37:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:37:40 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 19:37:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 19:37:50 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 19:37:55 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/09 19:37:55 App: TestOnUpdateN1QLOp got un-deployed
2021/12/09 19:37:55 No eventing-con process running
2021/12/09 19:37:55 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/09 19:37:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:38:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:38:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:38:19 Waiting for dst bucket item count to get to: 0 curr count: 4545
2021/12/09 19:38:24 Waiting for dst bucket item count to get to: 0 curr count: 2020
2021/12/09 19:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.43s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/09 19:38:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 19:38:34 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 19:38:34 No eventing-con process running
2021/12/09 19:38:34 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/09 19:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:38:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:38:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:38:47 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:38:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:38:48 Starting bucket ops to source bucket
2021/12/09 19:38:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:39:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:39:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:39:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:39:16 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 19:39:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:39:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:39:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:39:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 19:39:27 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 19:39:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 19:39:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/09 19:39:37 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/09 19:39:37 No eventing-con process running
2021/12/09 19:39:37 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/09 19:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:39:48 Waiting for dst bucket item count to get to: 0 curr count: 3954
2021/12/09 19:39:53 Waiting for dst bucket item count to get to: 0 curr count: 1454
2021/12/09 19:39:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:39:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:40:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:40:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.51s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/09 19:40:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 19:40:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 19:40:15 No eventing-con process running
2021/12/09 19:40:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 19:40:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:40:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:40:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:40:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:40:29 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:40:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 19:40:30 Starting bucket ops to source bucket
2021/12/09 19:40:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:40:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:40:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:40:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:40:58 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 19:40:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:40:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:40:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:41:03 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 19:41:08 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 19:41:13 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 19:41:18 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 19:41:18 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 19:41:18 No eventing-con process running
2021/12/09 19:41:18 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 19:41:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:41:30 Waiting for dst bucket item count to get to: 0 curr count: 2737
2021/12/09 19:41:35 Waiting for dst bucket item count to get to: 0 curr count: 2737
2021/12/09 19:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:41:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:41:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:41:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.52s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/09 19:41:57 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 19:41:57 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 19:41:57 No eventing-con process running
2021/12/09 19:41:57 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/09 19:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:42:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:42:10 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:42:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/09 19:42:11 Starting bucket ops to source bucket
2021/12/09 19:42:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:42:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:42:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:42:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:42:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:42:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:42:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:42:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 19:42:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 19:42:50 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 19:42:55 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/09 19:42:55 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/09 19:42:55 No eventing-con process running
2021/12/09 19:42:55 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/09 19:42:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:43:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:43:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:43:18 Waiting for dst bucket item count to get to: 0 curr count: 3712
2021/12/09 19:43:23 Waiting for dst bucket item count to get to: 0 curr count: 1212
2021/12/09 19:43:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (96.45s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/09 19:43:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 19:43:33 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 19:43:33 No eventing-con process running
2021/12/09 19:43:33 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/09 19:43:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:43:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:43:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:43:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:43:47 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:43:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/09 19:43:48 Starting bucket ops to source bucket
2021/12/09 19:43:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:44:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:44:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:44:11 Waiting for dst bucket item count to get to: 5000 curr count: 73
2021/12/09 19:44:16 Waiting for dst bucket item count to get to: 5000 curr count: 73
2021/12/09 19:44:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:44:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:44:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:44:21 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 19:44:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 19:44:32 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 19:44:37 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/09 19:44:37 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/09 19:44:37 No eventing-con process running
2021/12/09 19:44:37 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/09 19:44:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:44:48 Waiting for dst bucket item count to get to: 0 curr count: 3888
2021/12/09 19:44:53 Waiting for dst bucket item count to get to: 0 curr count: 1388
2021/12/09 19:44:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:44:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:45:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (101.98s)
=== RUN   TestOnDeleteBucketOp
2021/12/09 19:45:15 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 19:45:15 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 19:45:15 No eventing-con process running
2021/12/09 19:45:15 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/09 19:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:45:29 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:45:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 19:45:30 Starting bucket ops to source bucket
2021/12/09 19:45:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:45:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:45:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:45:56 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 19:45:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:45:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:45:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:46:01 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 19:46:06 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 19:46:11 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 19:46:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/09 19:46:16 App: TestOnDeleteBucketOp got un-deployed
2021/12/09 19:46:16 No eventing-con process running
2021/12/09 19:46:16 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/09 19:46:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:46:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:46:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:46:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.75s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/09 19:46:45 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 19:46:45 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 19:46:45 No eventing-con process running
2021/12/09 19:46:45 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/09 19:46:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:46:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:46:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:46:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:46:58 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:47:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/09 19:47:00 Starting bucket ops to source bucket
2021/12/09 19:47:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:47:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:47:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:47:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 19:47:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:47:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:47:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:47:31 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 19:47:36 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 19:47:41 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 19:47:46 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/09 19:47:46 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/09 19:47:46 No eventing-con process running
2021/12/09 19:47:46 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/09 19:47:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:47:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:48:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:48:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.94s)
=== RUN   TestTimerBucketOp
2021/12/09 19:48:15 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 19:48:15 App: TestTimerBucketOp got un-deployed
2021/12/09 19:48:15 No eventing-con process running
2021/12/09 19:48:15 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/09 19:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:48:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:48:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:48:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:48:28 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:48:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 19:48:29 Starting bucket ops to source bucket
2021/12/09 19:48:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:48:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:48:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:48:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:48:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:49:02 Waiting for dst bucket item count to get to: 5000 curr count: 50
2021/12/09 19:49:07 Waiting for dst bucket item count to get to: 5000 curr count: 1423
2021/12/09 19:49:12 Waiting for dst bucket item count to get to: 5000 curr count: 3873
2021/12/09 19:49:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:49:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:49:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:49:18 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 19:49:23 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 19:49:28 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 19:49:33 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/09 19:49:33 App: TestTimerBucketOp got un-deployed
2021/12/09 19:49:33 No eventing-con process running
2021/12/09 19:49:33 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/09 19:49:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:49:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:49:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:49:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:49:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:49:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.37s)
=== RUN   TestTimerInPastBucketOp
2021/12/09 19:50:06 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 19:50:06 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 19:50:06 No eventing-con process running
2021/12/09 19:50:06 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/09 19:50:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:50:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:50:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:50:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:50:20 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:50:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 19:50:21 Starting bucket ops to source bucket
2021/12/09 19:50:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:50:59 Waiting for dst bucket item count to get to: 5000 curr count: 2420
2021/12/09 19:51:04 Waiting for dst bucket item count to get to: 5000 curr count: 2420
2021/12/09 19:51:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:51:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:51:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:51:09 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 19:51:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 19:51:19 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 19:51:24 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/09 19:51:24 App: TestTimerInPastBucketOp got un-deployed
2021/12/09 19:51:24 No eventing-con process running
2021/12/09 19:51:24 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/09 19:51:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:51:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 19:51:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:51:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:51:47 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 19:51:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.29s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/09 19:51:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:51:57 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 19:51:57 No eventing-con process running
2021/12/09 19:51:57 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/09 19:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:52:11 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:52:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 19:52:12 Starting bucket ops to source bucket
2021/12/09 19:52:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:52:50 Waiting for dst bucket item count to get to: 5000 curr count: 3115
2021/12/09 19:52:55 Waiting for dst bucket item count to get to: 5000 curr count: 3115
2021/12/09 19:52:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:53:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 19:53:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 19:53:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:53:00 Undeploying app: bucket_op_with_timer
2021/12/09 19:53:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:53:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:53:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:53:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:53:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:53:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 19:53:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:53:20 No eventing-con process running
2021/12/09 19:53:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:53:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:54:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:54:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 19:54:05 Starting bucket ops to source bucket
2021/12/09 19:54:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:54:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:54:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:54:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:54:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:54:38 Waiting for dst bucket item count to get to: 5000 curr count: 30
2021/12/09 19:54:43 Waiting for dst bucket item count to get to: 5000 curr count: 30
2021/12/09 19:54:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:54:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 19:54:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 19:54:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:54:48 Undeploying app: bucket_op_with_timer
2021/12/09 19:54:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:54:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:54:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:55:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:55:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:55:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 19:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:55:08 No eventing-con process running
2021/12/09 19:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:55:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:55:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:55:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 19:55:54 Starting bucket ops to source bucket
2021/12/09 19:56:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:56:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 19:56:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:56:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 19:56:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 19:56:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:56:42 Undeploying app: bucket_op_with_timer
2021/12/09 19:56:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:56:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:56:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:56:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:57:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:57:02 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 19:57:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:57:02 No eventing-con process running
2021/12/09 19:57:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:57:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:57:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:57:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 19:57:47 Starting bucket ops to source bucket
2021/12/09 19:57:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:58:30 Waiting for dst bucket item count to get to: 5000 curr count: 3868
2021/12/09 19:58:35 Waiting for dst bucket item count to get to: 5000 curr count: 3868
2021/12/09 19:58:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 19:58:40 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 19:58:40 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 19:58:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 19:58:40 Undeploying app: bucket_op_with_timer
2021/12/09 19:58:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 19:58:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:58:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:58:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:59:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 19:59:00 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 19:59:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 19:59:00 No eventing-con process running
2021/12/09 19:59:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:59:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 19:59:44 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 19:59:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 19:59:45 Starting bucket ops to source bucket
2021/12/09 19:59:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 19:59:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:00:28 Waiting for dst bucket item count to get to: 5000 curr count: 4599
2021/12/09 20:00:33 Waiting for dst bucket item count to get to: 5000 curr count: 4599
2021/12/09 20:00:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:00:38 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/09 20:00:38 App: TestDeployUndeployLoopTimer got deployed
2021/12/09 20:00:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:00:38 Undeploying app: bucket_op_with_timer
2021/12/09 20:00:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:00:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 20:00:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 20:00:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 20:00:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/09 20:00:58 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/09 20:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:00:58 No eventing-con process running
2021/12/09 20:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:01:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:01:43 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (595.07s)
=== RUN   TestMultipleHandlers
2021/12/09 20:01:53 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 20:01:53 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 20:01:53 No eventing-con process running
2021/12/09 20:01:53 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/09 20:01:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:01:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:02:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:02:11 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 20:02:11 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 20:02:11 No eventing-con process running
2021/12/09 20:02:11 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/09 20:02:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:02:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:02:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:02:25 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:02:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 20:02:26 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:02:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 20:02:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 20:02:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 20:02:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 20:02:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/09 20:02:47 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/09 20:02:52 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 20:02:57 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 20:03:02 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/09 20:03:02 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/09 20:03:02 Starting bucket ops to source bucket
2021/12/09 20:03:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:03:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:03:12 Starting bucket ops to source bucket
2021/12/09 20:03:27 Waiting for dst bucket item count to get to: 10000 curr count: 6263
2021/12/09 20:03:32 Waiting for dst bucket item count to get to: 10000 curr count: 6263
2021/12/09 20:03:32 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:03:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:03:32 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:03:32 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:03:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:03:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 20:03:42 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 20:03:47 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 20:03:52 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/09 20:03:52 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/09 20:03:52 No eventing-con process running
2021/12/09 20:03:52 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/09 20:03:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:03:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:04:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:04:11 Waiting for dst bucket item count to get to: 0 curr count: 9653
2021/12/09 20:04:16 Waiting for dst bucket item count to get to: 0 curr count: 9653
2021/12/09 20:04:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:04:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:04:21 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 20:04:26 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 20:04:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 20:04:37 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/09 20:04:37 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/09 20:04:37 No eventing-con process running
2021/12/09 20:04:37 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/09 20:04:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:04:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:04:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (187.65s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/09 20:05:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 20:05:00 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 20:05:00 No eventing-con process running
2021/12/09 20:05:00 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/09 20:05:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:05:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:05:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:05:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:05:14 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:05:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 20:05:15 Starting bucket ops to source bucket
2021/12/09 20:05:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:05:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:05:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:05:38 Waiting for dst bucket item count to get to: 5000 curr count: 554
2021/12/09 20:05:43 Waiting for dst bucket item count to get to: 5000 curr count: 3054
2021/12/09 20:05:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:05:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:05:48 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 20:05:48 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 20:05:48 Pausing app: bucket_op_on_update
2021/12/09 20:05:48 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:05:53 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 20:05:53 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 20:05:53 Resuming app: bucket_op_on_update
2021/12/09 20:05:54 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:05:54 Starting bucket ops to source bucket
2021/12/09 20:06:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:06:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:06:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:06:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:06:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:06:17 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 20:06:17 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 20:06:17 Pausing app: bucket_op_on_update
2021/12/09 20:06:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:06:22 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 20:06:22 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 20:06:22 Resuming app: bucket_op_on_update
2021/12/09 20:06:23 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:06:23 Starting bucket ops to source bucket
2021/12/09 20:06:31 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:06:36 Waiting for dst bucket item count to get to: 15000 curr count: 10319
2021/12/09 20:06:41 Waiting for dst bucket item count to get to: 15000 curr count: 10319
2021/12/09 20:06:41 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 20:06:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:06:46 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 20:06:46 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 20:06:46 Pausing app: bucket_op_on_update
2021/12/09 20:06:46 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:06:51 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 20:06:51 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 20:06:51 Resuming app: bucket_op_on_update
2021/12/09 20:06:53 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:06:53 Starting bucket ops to source bucket
2021/12/09 20:07:00 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:07:05 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:07:10 Waiting for dst bucket item count to get to: 20000 curr count: 15561
2021/12/09 20:07:15 Waiting for dst bucket item count to get to: 20000 curr count: 15561
2021/12/09 20:07:15 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 20:07:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:07:20 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 20:07:20 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 20:07:20 Pausing app: bucket_op_on_update
2021/12/09 20:07:21 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:07:26 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 20:07:26 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 20:07:26 Resuming app: bucket_op_on_update
2021/12/09 20:07:27 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:07:27 Starting bucket ops to source bucket
2021/12/09 20:07:35 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:07:40 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:07:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:07:45 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 20:07:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:07:50 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/09 20:07:50 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/09 20:07:50 Pausing app: bucket_op_on_update
2021/12/09 20:07:50 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:07:55 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/09 20:07:55 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/09 20:07:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:08:00 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 20:08:05 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 20:08:10 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 20:08:15 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/09 20:08:15 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/09 20:08:15 No eventing-con process running
2021/12/09 20:08:15 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/09 20:08:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:08:27 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 20:08:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:08:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:08:38 Waiting for dst bucket item count to get to: 0 curr count: 21767
2021/12/09 20:08:44 Waiting for dst bucket item count to get to: 0 curr count: 9267
2021/12/09 20:08:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.31s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/09 20:08:54 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 20:08:54 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 20:08:54 No eventing-con process running
2021/12/09 20:08:54 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/09 20:08:55 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:09:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:09:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:09:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:09:07 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:09:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 20:09:08 Starting bucket ops to source bucket
2021/12/09 20:09:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:09:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:09:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:09:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:09:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:09:41 Waiting for dst bucket item count to get to: 5000 curr count: 4
2021/12/09 20:09:46 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/09 20:09:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:09:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:09:51 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 20:09:51 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 20:09:51 Pausing app: bucket_op_with_timer
2021/12/09 20:09:51 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:09:56 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 20:09:56 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 20:09:56 Resuming app: bucket_op_with_timer
2021/12/09 20:09:57 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:09:57 Starting bucket ops to source bucket
2021/12/09 20:10:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:10:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:10:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:10:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:10:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:10:30 Waiting for dst bucket item count to get to: 10000 curr count: 5864
2021/12/09 20:10:35 Waiting for dst bucket item count to get to: 10000 curr count: 8364
2021/12/09 20:10:35 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:10:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:10:40 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 20:10:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 20:10:40 Pausing app: bucket_op_with_timer
2021/12/09 20:10:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:10:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 20:10:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 20:10:46 Resuming app: bucket_op_with_timer
2021/12/09 20:10:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:10:47 Starting bucket ops to source bucket
2021/12/09 20:10:55 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:11:00 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:11:05 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:11:10 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:11:15 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/09 20:11:20 Waiting for dst bucket item count to get to: 15000 curr count: 11893
2021/12/09 20:11:25 Waiting for dst bucket item count to get to: 15000 curr count: 14393
2021/12/09 20:11:25 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/09 20:11:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:11:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 20:11:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 20:11:30 Pausing app: bucket_op_with_timer
2021/12/09 20:11:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:11:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 20:11:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 20:11:35 Resuming app: bucket_op_with_timer
2021/12/09 20:11:36 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:11:36 Starting bucket ops to source bucket
2021/12/09 20:11:44 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:11:49 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:11:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:11:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:12:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/09 20:12:09 Waiting for dst bucket item count to get to: 20000 curr count: 17577
2021/12/09 20:12:14 Waiting for dst bucket item count to get to: 20000 curr count: 17577
2021/12/09 20:12:14 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 20:12:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:12:19 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 20:12:19 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 20:12:19 Pausing app: bucket_op_with_timer
2021/12/09 20:12:19 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:12:24 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 20:12:24 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 20:12:24 Resuming app: bucket_op_with_timer
2021/12/09 20:12:25 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:12:25 Starting bucket ops to source bucket
2021/12/09 20:12:33 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:12:38 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:12:43 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:12:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:12:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/09 20:12:58 Waiting for dst bucket item count to get to: 25000 curr count: 23349
2021/12/09 20:13:03 Waiting for dst bucket item count to get to: 25000 curr count: 23349
2021/12/09 20:13:03 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 20:13:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:13:08 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/09 20:13:08 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/09 20:13:08 Pausing app: bucket_op_with_timer
2021/12/09 20:13:08 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:13:13 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/09 20:13:13 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/09 20:13:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:13:18 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 20:13:23 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 20:13:29 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 20:13:34 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/09 20:13:34 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/09 20:13:34 No eventing-con process running
2021/12/09 20:13:34 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/09 20:13:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:13:45 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 20:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:13:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:13:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 20:13:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (313.54s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/09 20:14:07 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 20:14:07 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 20:14:07 No eventing-con process running
2021/12/09 20:14:07 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/09 20:14:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:14:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:14:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:14:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:14:21 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:14:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/09 20:14:27 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 20:14:32 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 20:14:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 20:14:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/09 20:14:42 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/09 20:14:42 Starting bucket ops to source bucket
2021/12/09 20:14:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:14:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:15:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:15:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:15:11 Waiting for dst bucket item count to get to: 5000 curr count: 4683
2021/12/09 20:15:16 Waiting for dst bucket item count to get to: 5000 curr count: 4683
2021/12/09 20:15:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:15:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:15:16 Pausing app: bucket_op_with_timer
2021/12/09 20:15:16 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:15:21 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/09 20:15:21 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/09 20:15:21 Resuming app: bucket_op_with_timer
2021/12/09 20:15:22 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:15:22 Starting bucket ops to source bucket
2021/12/09 20:15:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:15:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:15:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:15:49 Waiting for dst bucket item count to get to: 10000 curr count: 5419
2021/12/09 20:15:54 Waiting for dst bucket item count to get to: 10000 curr count: 5419
2021/12/09 20:15:59 Waiting for dst bucket item count to get to: 10000 curr count: 9231
2021/12/09 20:16:04 Waiting for dst bucket item count to get to: 10000 curr count: 9231
2021/12/09 20:16:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:16:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:16:09 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 20:16:14 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 20:16:19 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 20:16:24 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/09 20:16:24 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/09 20:16:24 No eventing-con process running
2021/12/09 20:16:24 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/09 20:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:16:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 20:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:16:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:16:48 Waiting for dst bucket item count to get to: 0 curr count: 6829
2021/12/09 20:16:53 Waiting for dst bucket item count to get to: 0 curr count: 1829
2021/12/09 20:16:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (175.91s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/09 20:17:03 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 20:17:03 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 20:17:03 No eventing-con process running
2021/12/09 20:17:03 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/09 20:17:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:17:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:17:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:17:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:17:17 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:17:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 20:17:23 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 20:17:28 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 20:17:33 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 20:17:38 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/09 20:17:38 App: TestPauseResumeWithEventingReb got deployed
2021/12/09 20:17:38 Starting bucket ops to source bucket
2021/12/09 20:17:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:17:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:17:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:18:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:18:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:18:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:18:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:18:07 Pausing app: bucket_op_with_timer
2021/12/09 20:18:07 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:18:12 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/09 20:18:12 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/09 20:18:12 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 20:18:12 Removing node(s): [] from the cluster
2021/12/09 20:18:17 Rebalance progress: 29.16666666666666
2021/12/09 20:18:22 Rebalance progress: 29.16666666666666
2021/12/09 20:18:27 Rebalance progress: 100
2021/12/09 20:18:27 Resuming app: bucket_op_with_timer
2021/12/09 20:18:28 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:18:28 Starting bucket ops to source bucket
2021/12/09 20:18:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:18:45 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:18:50 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:18:55 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:19:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/09 20:19:05 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/09 20:19:05 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:19:05 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 20:19:10 Rebalance progress: 29.16666666666666
2021/12/09 20:19:15 Rebalance progress: 29.16666666666666
2021/12/09 20:19:20 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 20:19:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:19:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:19:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:20:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:20:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:20:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:20:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:21:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:21:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:21:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 20:21:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:64} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6543c9b5aaeee5803a2b3c216b0aded6
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
Producer node: 172.31.2.134:9300	Node UUID: 9b64e2177e1c28c0c5298a0f21ce76e9
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
Producer node: 127.0.0.1:9303	Node UUID: 36d41f79e0f8adf413a7f4ca5b36e21a
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]


2021/12/09 20:22:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:22:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:22:16 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 20:22:21 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 20:22:26 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 20:22:31 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/09 20:22:31 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/09 20:22:31 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:22:31 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/09 20:22:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:22:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 20:22:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:22:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:22:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:22:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:23:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:23:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (371.52s)
=== RUN   TestBucketDeleteAfterPause
2021/12/09 20:23:14 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 20:23:14 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 20:23:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:23:15 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/09 20:23:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:23:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:23:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:23:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:23:28 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:23:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/09 20:23:34 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 20:23:39 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 20:23:44 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 20:23:49 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/09 20:23:49 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/09 20:23:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:23:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:24:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 18
2021/12/09 20:24:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2518
2021/12/09 20:24:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:24:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:24:09 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/09 20:24:09 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:24:14 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/09 20:24:14 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/09 20:24:14 handler_test.go:716: Deleting source bucket: default
2021/12/09 20:24:21 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/09 20:24:21 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/09 20:24:26 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/09 20:24:26 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/09 20:24:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:24:26 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/09 20:24:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:24:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:24:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:24:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:24:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4964
2021/12/09 20:24:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2464
2021/12/09 20:24:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.41s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/09 20:25:04 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 20:25:04 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 20:25:04 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:25:04 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/09 20:25:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:25:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:25:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:25:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:25:17 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:25:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 20:25:23 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 20:25:28 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 20:25:33 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 20:25:38 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/09 20:25:38 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/09 20:25:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:26:13 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/09 20:26:13 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:26:18 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/09 20:26:18 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/09 20:29:18 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/09 20:29:18 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/09 20:29:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/09 20:29:24 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 20:29:24 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:29:24 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/09 20:29:29 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 20:29:29 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:29:29 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3886 expected 5000
2021/12/09 20:29:34 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/09 20:29:34 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:29:34 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/09 20:29:34 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/09 20:29:35 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:29:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 20:29:50 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 20:29:55 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/09 20:29:55 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/09 20:29:55 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:29:55 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/09 20:29:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:30:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:30:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:30:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:30:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (318.77s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/09 20:30:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 20:30:23 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 20:30:23 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:30:23 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/09 20:30:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:30:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:30:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:30:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:30:36 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:30:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 20:30:42 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:30:47 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:30:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:30:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:30:57 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 20:30:57 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:31:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:31:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1534
2021/12/09 20:31:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1534
2021/12/09 20:31:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:31:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:31:16 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 20:31:16 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:31:46 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/09 20:31:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 20:31:52 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:31:57 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:32:02 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:32:07 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/09 20:32:07 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/09 20:32:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:32:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:32:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 20:32:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:32:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:32:25 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/09 20:32:25 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:32:35 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 20:32:40 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 20:32:45 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/09 20:32:45 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/09 20:32:45 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:32:45 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/09 20:32:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:32:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:32:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:32:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.37s)
=== RUN   TestCPPWorkerCleanup
2021/12/09 20:33:08 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 20:33:08 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 20:33:08 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:33:08 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/09 20:33:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:33:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:33:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:33:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:33:21 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:33:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/09 20:33:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:33:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:33:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:33:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:33:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:33:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3148
2021/12/09 20:33:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3148
2021/12/09 20:33:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:33:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:33:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:33:59 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 20:34:05 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 20:34:10 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 20:34:15 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/09 20:34:15 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/09 20:34:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:34:15 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/09 20:34:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:34:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:34:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:34:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:34:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 20:34:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (129.58s)
=== RUN   TestWithUserXattrs
2021/12/09 20:35:18 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 20:35:18 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 20:35:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:35:18 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/09 20:35:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:35:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:35:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:35:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:35:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:35:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:35:34 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:35:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/09 20:35:41 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 20:35:46 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 20:35:51 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 20:35:56 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/09 20:35:56 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/09 20:35:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:36:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:36:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3080
2021/12/09 20:36:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3080
2021/12/09 20:36:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:36:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:36:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:36:20 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 20:36:25 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 20:36:30 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 20:36:35 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/09 20:36:35 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/09 20:36:35 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:36:35 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/09 20:36:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:36:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:36:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:36:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:36:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3575
2021/12/09 20:37:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1075
2021/12/09 20:37:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (115.42s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/09 20:37:13 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:37:13 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 20:37:13 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:37:13 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/09 20:37:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:37:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:37:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:37:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:37:26 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:37:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/09 20:37:32 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 20:37:37 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 20:37:42 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 20:37:47 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/09 20:37:47 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/09 20:37:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:37:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:38:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1289
2021/12/09 20:38:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1289
2021/12/09 20:38:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:38:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:38:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:38:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1649
2021/12/09 20:38:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1649
2021/12/09 20:38:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:38:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:38:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:38:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:38:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:39:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 13768
2021/12/09 20:39:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 20057
2021/12/09 20:39:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/09 20:39:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:39:07 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:39:12 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:39:17 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:39:22 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:39:27 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:39:27 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 20:39:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:39:32 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/09 20:39:32 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/09 20:39:32 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:39:32 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/09 20:39:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:39:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/09 20:39:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/09 20:39:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:39:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:40:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 18751
2021/12/09 20:40:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 18751
2021/12/09 20:40:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (182.24s)
=== RUN   TestMetaBucketDelete
2021/12/09 20:40:15 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 20:40:15 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 20:40:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:40:15 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/09 20:40:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:40:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:40:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:40:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:40:28 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:40:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 20:40:34 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 20:40:39 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 20:40:44 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 20:40:49 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/09 20:40:49 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/09 20:40:49 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/09 20:40:50 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/09 20:40:55 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 20:40:55 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 20:41:11 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/09 20:41:11 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/09 20:41:11 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:41:11 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/09 20:41:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:41:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:41:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:41:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.33s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/09 20:41:34 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:41:34 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:41:34 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:41:34 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 20:41:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:41:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:41:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:41:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:41:47 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:41:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 20:42:07 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/09 20:42:07 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/09 20:42:22 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:42:22 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:42:37 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:42:37 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:42:37 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:42:37 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 20:42:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:42:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:42:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:42:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.63s)
=== RUN   TestSourceBucketDelete
2021/12/09 20:43:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 20:43:00 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 20:43:00 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:43:00 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/09 20:43:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:43:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:43:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:43:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:43:13 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:43:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/09 20:43:19 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 20:43:24 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 20:43:29 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 20:43:34 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/09 20:43:34 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/09 20:43:34 handler_test.go:945: Deleting source bucket: default
2021/12/09 20:43:36 handler_test.go:947: Deleted source bucket: default
2021/12/09 20:43:41 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 20:43:41 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 20:43:56 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/09 20:43:56 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/09 20:43:56 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:43:56 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/09 20:43:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:44:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:44:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:44:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.92s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/09 20:44:19 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:44:19 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:44:19 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:44:19 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 20:44:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:44:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:44:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:44:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:44:32 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:44:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 20:44:52 handler_test.go:966: Deleting source bucket: default
2021/12/09 20:44:54 handler_test.go:968: Deleted source bucket: default
2021/12/09 20:45:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:45:09 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:45:24 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:45:24 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:45:24 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:45:24 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 20:45:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:45:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:45:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:45:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.55s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/09 20:45:47 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:45:47 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:45:47 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:45:47 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/09 20:45:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:45:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:45:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:46:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:46:00 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:46:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 20:46:20 handler_test.go:988: Deleting source bucket: default
2021/12/09 20:46:21 handler_test.go:990: Deleted source bucket: default
2021/12/09 20:46:21 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/09 20:46:21 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/09 20:46:36 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:46:36 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:46:52 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/09 20:46:52 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/09 20:46:52 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:46:52 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/09 20:46:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:46:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:47:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:47:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (87.94s)
=== RUN   TestUndeployDuringBootstrap
2021/12/09 20:47:15 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 20:47:15 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 20:47:15 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:47:15 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/09 20:47:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:47:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:47:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:47:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:47:28 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:47:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 20:47:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:47:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:47:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:47:41 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 20:47:41 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/09 20:47:41 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 20:47:46 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/09 20:47:51 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/09 20:47:56 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 20:48:01 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 20:48:06 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 20:48:06 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 20:48:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:48:31 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/09 20:48:31 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/09 20:48:31 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:48:31 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/09 20:48:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:48:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:48:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:48:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:48:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:48:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3878
2021/12/09 20:49:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1378
2021/12/09 20:49:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.36s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/09 20:49:09 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:49:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/09 20:49:15 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 20:49:20 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 20:49:25 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 20:49:30 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/09 20:49:30 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/09 20:49:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/09 20:49:30 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:49:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/09 20:49:35 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/09 20:49:35 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/09 20:49:35 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:49:35 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/09 20:49:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:49:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:49:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:49:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.47s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/09 20:49:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:49:59 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 20:49:59 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:49:59 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/09 20:50:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:50:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:50:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:50:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:50:12 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:50:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/09 20:50:18 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 20:50:23 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 20:50:28 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 20:50:33 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/09 20:50:33 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/09 20:50:33 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:51:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:51:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:51:13 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:51:18 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:51:23 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:51:23 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 20:51:23 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:53:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/09 20:53:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/09 20:53:08 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:53:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/09 20:53:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:53:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 20:53:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 20:53:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:53:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:53:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:53:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (227.98s)
=== RUN   TestUndeployWithKVFailover
2021/12/09 20:53:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 20:53:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/09 20:53:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 20:53:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/09 20:53:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:53:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:53:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:54:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:54:00 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:54:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/09 20:54:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 20:54:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 20:54:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2452
2021/12/09 20:54:43 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2452
2021/12/09 20:54:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 20:54:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 20:54:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 20:54:48 Rebalance progress: 29.16666666666666
2021/12/09 20:54:53 Rebalance progress: 29.16666666666666
2021/12/09 20:54:58 Rebalance progress: 100
2021/12/09 20:54:58 Failing over: [127.0.0.1:19003]
2021/12/09 20:55:08 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/09 20:56:08 Removing node(s): [] from the cluster
2021/12/09 20:56:13 Rebalance progress: 29.16666666666666
2021/12/09 20:56:18 Rebalance progress: 29.16666666666666
2021/12/09 20:56:23 Rebalance progress: 100
2021/12/09 20:56:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 20:56:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 20:56:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:56:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/09 20:56:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/09 20:56:33 No eventing-con process running
2021/12/09 20:56:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/09 20:56:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:56:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:56:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:56:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 20:56:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (199.94s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/09 20:57:06 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 20:57:06 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 20:57:06 No eventing-con process running
2021/12/09 20:57:06 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/09 20:57:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:57:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:57:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:57:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:57:20 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/09 20:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/09 20:57:26 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 20:57:31 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 20:57:36 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 20:57:41 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/09 20:57:41 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/09 20:57:41 Starting bucket ops to source bucket
2021/12/09 20:58:01 Waiting for dst bucket item count to get to: 20000 curr count: 9766
2021/12/09 20:58:01 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/09 20:58:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:58:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:58:13 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/09 20:58:18 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 20:58:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:58:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:58:30 Waiting for dst bucket item count to get to: 0 curr count: 14427
2021/12/09 20:58:35 Waiting for dst bucket item count to get to: 0 curr count: 14427
2021/12/09 20:58:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:58:35 Starting bucket ops to source bucket
2021/12/09 20:58:48 Waiting for dst bucket item count to get to: 10000 curr count: 3778
2021/12/09 20:58:53 Waiting for dst bucket item count to get to: 10000 curr count: 3778
2021/12/09 20:58:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 20:58:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 20:58:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 20:58:58 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 20:59:03 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 20:59:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 20:59:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/09 20:59:13 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/09 20:59:13 No eventing-con process running
2021/12/09 20:59:13 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/09 20:59:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:59:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 20:59:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:59:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:59:36 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 20:59:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (159.72s)
=== RUN   TestTimerOverwrite
2021/12/09 20:59:46 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 20:59:46 App: TestTimerOverwrite got un-deployed
2021/12/09 20:59:46 No eventing-con process running
2021/12/09 20:59:46 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/09 20:59:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:59:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:59:54 Context: Bucket flush, response code: 200 dump: 
2021/12/09 20:59:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 20:59:59 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:00:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/09 21:00:05 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 21:00:10 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 21:00:15 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 21:00:20 Waiting for app: TestTimerOverwrite to get deployed
2021/12/09 21:00:20 App: TestTimerOverwrite got deployed
2021/12/09 21:00:30 Starting bucket ops to source bucket
2021/12/09 21:00:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:00:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:00:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:00:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:01:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 21:01:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:01:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:01:50 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 21:01:55 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 21:02:00 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 21:02:05 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/09 21:02:05 App: TestTimerOverwrite got un-deployed
2021/12/09 21:02:05 No eventing-con process running
2021/12/09 21:02:05 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/09 21:02:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:02:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:02:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:02:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:02:28 Waiting for dst bucket item count to get to: 0 curr count: 3239
2021/12/09 21:02:33 Waiting for dst bucket item count to get to: 0 curr count: 710
2021/12/09 21:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (176.75s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/09 21:02:38 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/09 21:02:38 Removing node(s): [] from the cluster
2021/12/09 21:02:43 Rebalance progress: 100
2021/12/09 21:02:43 Failing over: [127.0.0.1:19003]
2021/12/09 21:02:48 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 21:02:48 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 21:02:48 No eventing-con process running
2021/12/09 21:02:48 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 21:02:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:02:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:02:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:03:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:03:01 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:03:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/09 21:03:03 Starting bucket ops to source bucket
2021/12/09 21:03:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:03:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:03:13 Context: Post to set retry counter, response code: 200 dump: 
2021/12/09 21:03:13 set retry counter, response &{[] }
2021/12/09 21:04:18 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 21:04:18 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 21:04:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:04:18 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/09 21:04:23 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/09 21:04:23 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/09 21:04:23 No eventing-con process running
2021/12/09 21:04:23 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/09 21:04:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:04:34 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:04:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 21:04:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:04:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:04:51 Waiting for dst bucket item count to get to: 0 curr count: 1826
2021/12/09 21:04:56 Waiting for dst bucket item count to get to: 0 curr count: 1826
2021/12/09 21:04:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:04:56 Removing node(s): [] from the cluster
2021/12/09 21:05:01 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (147.85s)
=== RUN   TestOnUpdateSrcMutation
2021/12/09 21:05:11 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 21:05:11 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 21:05:11 No eventing-con process running
2021/12/09 21:05:11 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/09 21:05:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:05:19 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:05:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:05:24 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:05:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 21:05:25 Starting bucket ops to source bucket
2021/12/09 21:05:52 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 21:05:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:05:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:05:57 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 21:06:02 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 21:06:07 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 21:06:12 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/09 21:06:12 App: TestOnUpdateSrcMutation got un-deployed
2021/12/09 21:06:12 No eventing-con process running
2021/12/09 21:06:12 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/09 21:06:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:06:23 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 21:06:28 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:06:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:06:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.41s)
=== RUN   TestOnDeleteSrcMutation
2021/12/09 21:06:45 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 21:06:45 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 21:06:45 No eventing-con process running
2021/12/09 21:06:45 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/09 21:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:06:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:06:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:06:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:06:58 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:06:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/09 21:06:59 Starting bucket ops to source bucket
2021/12/09 21:07:28 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 21:07:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:07:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:07:33 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 21:07:38 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 21:07:43 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 21:07:48 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/09 21:07:48 App: TestOnDeleteSrcMutation got un-deployed
2021/12/09 21:07:48 No eventing-con process running
2021/12/09 21:07:48 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/09 21:07:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:07:59 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 21:08:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 21:08:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:08:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.08s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/09 21:08:21 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 21:08:21 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 21:08:21 No eventing-con process running
2021/12/09 21:08:21 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/09 21:08:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:08:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:08:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:08:34 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:08:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 21:08:35 Starting bucket ops to source bucket
2021/12/09 21:09:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 21:09:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:09:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:09:17 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 21:09:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 21:09:27 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 21:09:32 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/09 21:09:32 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/09 21:09:33 No eventing-con process running
2021/12/09 21:09:33 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/09 21:09:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:09:44 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 21:09:49 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:09:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:09:51 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:09:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (104.29s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/09 21:10:06 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 21:10:06 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 21:10:06 No eventing-con process running
2021/12/09 21:10:06 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/09 21:10:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:10:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:10:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:10:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:10:19 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:10:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/09 21:10:20 Starting bucket ops to source bucket
2021/12/09 21:11:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 21:11:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:11:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:11:08 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 21:11:14 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 21:11:19 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 21:11:24 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/09 21:11:24 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/09 21:11:24 No eventing-con process running
2021/12/09 21:11:24 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/09 21:11:25 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:11:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:11:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:11:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:11:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.04s)
=== RUN   TestInterHandlerRecursion
2021/12/09 21:11:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:11:52 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 21:11:52 No eventing-con process running
2021/12/09 21:11:52 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 21:11:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:11:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:12:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:12:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:12:10 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:12:10 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 21:12:10 No eventing-con process running
2021/12/09 21:12:10 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 21:12:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:12:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:12:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:12:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:12:23 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 21:12:24 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:12:29 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:12:34 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:12:39 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:12:44 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:12:44 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 21:12:44 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:12:44 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/09 21:12:44 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/09 21:12:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:12:49 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:12:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:12:59 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:13:04 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:13:04 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 21:13:04 No eventing-con process running
2021/12/09 21:13:04 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 21:13:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:13:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:13:12 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:13:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:13:22 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:13:22 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 21:13:22 No eventing-con process running
2021/12/09 21:13:22 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 21:13:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:13:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.65s)
=== RUN   TestInterBucketRecursion
2021/12/09 21:13:40 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:13:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:13:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:13:41 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:13:41 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 21:13:46 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 21:13:51 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 21:13:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 21:14:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/09 21:14:01 App: TestInterBucketRecursion_function1 got deployed
2021/12/09 21:14:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:14:07 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:14:12 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:14:17 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:14:22 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:14:22 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/09 21:14:22 No eventing-con process running
2021/12/09 21:14:22 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 21:14:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:14:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:14:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:14:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:14:40 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/09 21:14:40 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/09 21:14:40 No eventing-con process running
2021/12/09 21:14:40 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/09 21:14:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:14:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:14:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.46s)
=== RUN   TestLargeHandler
2021/12/09 21:14:53 Changed force_compress value to false
2021/12/09 21:14:53 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:14:53 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/09 21:14:53 Changed force_compress value to true
2021/12/09 21:14:53 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:14:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/09 21:15:00 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 21:15:05 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 21:15:10 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 21:15:15 Waiting for app: TestLargeHandler_function to get deployed
2021/12/09 21:15:15 App: TestLargeHandler_function got deployed
2021/12/09 21:15:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:15:20 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 21:15:25 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 21:15:30 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 21:15:35 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/09 21:15:35 App: TestLargeHandler_function got un-deployed
2021/12/09 21:15:35 No eventing-con process running
2021/12/09 21:15:35 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/09 21:15:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:15:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:15:43 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:15:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.34s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/09 21:15:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:15:58 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 21:15:58 No eventing-con process running
2021/12/09 21:15:58 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/09 21:15:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:16:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:16:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:16:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:16:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:16:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 21:16:16 No eventing-con process running
2021/12/09 21:16:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/09 21:16:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:16:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:16:24 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:16:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:16:29 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:16:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/09 21:16:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:16:35 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:16:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:16:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:16:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/09 21:16:50 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/09 21:16:50 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:16:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/09 21:16:52 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:16:57 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 21:17:02 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 21:17:07 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 21:17:12 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/09 21:17:12 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/09 21:17:12 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/09 21:17:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:17:17 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:17:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:17:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:17:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/09 21:17:32 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/09 21:17:32 No eventing-con process running
2021/12/09 21:17:32 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/09 21:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:17:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:17:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:17:51 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:17:56 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:18:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:18:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/09 21:18:06 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/09 21:18:06 No eventing-con process running
2021/12/09 21:18:06 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/09 21:18:07 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:18:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:18:14 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.07s)
=== RUN   TestAllowInterBucketRecursion
2021/12/09 21:18:24 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:18:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:18:25 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:18:30 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 21:18:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 21:18:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 21:18:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/09 21:18:45 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/09 21:18:45 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:18:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 21:18:46 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:18:51 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 21:18:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 21:19:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 21:19:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/09 21:19:06 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/09 21:19:06 TestAllowInterBucketRecursion_function2 is deployed
2021/12/09 21:19:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:19:12 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:19:17 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:19:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:19:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/09 21:19:27 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/09 21:19:27 No eventing-con process running
2021/12/09 21:19:27 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/09 21:19:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:19:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:19:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:19:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:19:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:19:46 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 21:19:51 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 21:19:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 21:20:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/09 21:20:01 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/09 21:20:01 No eventing-con process running
2021/12/09 21:20:01 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/09 21:20:02 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:20:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (114.88s)
=== RUN   TestInterBucketRecursion2
2021/12/09 21:20:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:26 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:20:43 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:20:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:20:50 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 21:20:55 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 21:21:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 21:21:05 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/09 21:21:05 App: TestInterBucketRecursion2_function1 got deployed
2021/12/09 21:21:05 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:21:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 21:21:11 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 21:21:16 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 21:21:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 21:21:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/09 21:21:26 App: TestInterBucketRecursion2_function2 got deployed
2021/12/09 21:21:26 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:21:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 21:21:32 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 21:21:37 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 21:21:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 21:21:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/09 21:21:47 App: TestInterBucketRecursion2_function3 got deployed
2021/12/09 21:21:47 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:21:47 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/09 21:21:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:21:52 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 21:21:57 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 21:22:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 21:22:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/09 21:22:07 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/09 21:22:07 No eventing-con process running
2021/12/09 21:22:07 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/09 21:22:09 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:22:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:22:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:22:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:22:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:22:26 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 21:22:31 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 21:22:36 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 21:22:41 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/09 21:22:41 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/09 21:22:41 No eventing-con process running
2021/12/09 21:22:41 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/09 21:22:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:22:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:22:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:22:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:22:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:22:59 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 21:23:04 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 21:23:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 21:23:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/09 21:23:14 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/09 21:23:14 No eventing-con process running
2021/12/09 21:23:14 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/09 21:23:15 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:23:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:23:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:23:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:23:32 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/09 21:23:32 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/09 21:23:32 No eventing-con process running
2021/12/09 21:23:32 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/09 21:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:23:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:23:40 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:23:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.32s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/09 21:23:52 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 21:23:57 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/09 21:24:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:24:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:24:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:24:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:25:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:25:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:25:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:25:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:26:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:26:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:26:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:26:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:27:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:27:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:27:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:27:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:28:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:28:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:28:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:28:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:29:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:29:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:29:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/09 21:29:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:42} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/09 21:30:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:30:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 21:30:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 21:30:06 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 21:30:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:30:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 21:30:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 21:30:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/09 21:30:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:30:16 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 21:30:16 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 21:30:16 eventingSetup.go:46: No eventing-con process running
2021/12/09 21:30:16 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/09 21:30:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:30:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:30:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:30:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:30:34 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:30:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 21:30:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 21:30:45 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 21:30:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 21:30:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/09 21:30:55 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6543c9b5aaeee5803a2b3c216b0aded6
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 9b64e2177e1c28c0c5298a0f21ce76e9
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 36d41f79e0f8adf413a7f4ca5b36e21a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 21:30:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:30:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 21:31:00 Rebalance progress: 29.16666666666666
2021/12/09 21:31:05 Rebalance progress: 29.16666666666666
2021/12/09 21:31:10 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6543c9b5aaeee5803a2b3c216b0aded6
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: 9b64e2177e1c28c0c5298a0f21ce76e9
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: 36d41f79e0f8adf413a7f4ca5b36e21a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/09 21:31:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:31:11 handler_test.go:1674: Deleting source bucket: default
2021/12/09 21:31:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 21:31:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 21:31:18 Rebalance progress: 29.16666666666666
2021/12/09 21:31:22 Rebalance progress: 29.16666666666666
2021/12/09 21:31:27 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: stopped
	len: 121
	vb list dump: [9, 16, 22, 26, 40, 49, 64, 74-75, 86, 98, 101, 106, 111, 131, 133, 140, 142, 150, 161, 166, 200, 203-204, 206, 215, 224, 231, 236, 258, 278, 282, 285, 291, 293, 301, 305, 313, 320, 328, 345, 350, 374, 386, 388, 396, 399, 401, 406, 409, 415, 418, 422, 456, 461-462, 465, 467-468, 478, 483, 486-488, 490, 497, 500, 504, 571, 600, 606, 618, 623, 627-628, 635, 646, 658, 666, 669, 683, 690, 702, 707, 743, 747, 760-761, 775, 780, 784-786, 800, 803, 807, 819, 823, 826, 828, 836, 838, 842, 852, 871, 887, 889, 901, 904, 913, 926, 932, 950, 970-971, 975, 978, 990-991, 1014, 1022]
	stream status: running
	len: 903
	vb list dump: [0-8, 10-15, 17-21, 23-25, 27-39, 41-48, 50-63, 65-73, 76-85, 87-97, 99-100, 102-105, 107-110, 112-130, 132, 134-139, 141, 143-149, 151-160, 162-165, 167-199, 201-202, 205, 207-214, 216-223, 225-230, 232-235, 237-257, 259-277, 279-281, 283-284, 286-290, 292, 294-300, 302-304, 306-312, 314-319, 321-327, 329-344, 346-349, 351-373, 375-385, 387, 389-395, 397-398, 400, 402-405, 407-408, 410-414, 416-417, 419-421, 423-455, 457-460, 463-464, 466, 469-477, 479-482, 484-485, 489, 491-496, 498-499, 501-503, 505-570, 572-599, 601-605, 607-617, 619-622, 624-626, 629-634, 636-645, 647-657, 659-665, 667-668, 670-682, 684-689, 691-701, 703-706, 708-742, 744-746, 748-759, 762-774, 776-779, 781-783, 787-799, 801-802, 804-806, 808-818, 820-822, 824-825, 827, 829-835, 837, 839-841, 843-851, 853-870, 872-886, 888, 890-900, 902-903, 905-912, 914-925, 927-931, 933-949, 951-969, 972-974, 976-977, 979-989, 992-1013, 1015-1021, 1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6543c9b5aaeee5803a2b3c216b0aded6
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-230, 232-235, 237-257, 259-277, 279-281, 283-284, 286-290, 292, 294-300, 302-304, 306-312, 314-319, 321-327, 329-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-8, 10-15, 17-21, 23-25, 27-39, 41-48, 50-63, 65-73, 76-85, 87-97, 99-100, 102-105, 107-110, 112-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-130, 132, 134-139, 141, 143-149, 151-160, 162-165, 167-199, 201-202, 205, 207-214, 216-223, 225-227]
Producer node: 172.31.2.134:9300	Node UUID: 9b64e2177e1c28c0c5298a0f21ce76e9
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-912, 914-925, 927-931, 933-949, 951-969, 972-974, 976-977, 979-989, 992-1013, 1015-1021, 1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [684-689, 691-701, 703-706, 708-742, 744-746, 748-759, 762-774, 776-779, 781-783, 787-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [797-799, 801-802, 804-806, 808-818, 820-822, 824-825, 827, 829-835, 837, 839-841, 843-851, 853-870, 872-886, 888, 890-900, 902-903, 905-910]
Producer node: 127.0.0.1:9303	Node UUID: 36d41f79e0f8adf413a7f4ca5b36e21a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-344, 346-349, 351-373, 375-385, 387, 389-395, 397-398, 400, 402-405, 407-408, 410-414, 416-417, 419-421, 423-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [457-460, 463-464, 466, 469-477, 479-482, 484-485, 489, 491-496, 498-499, 501-503, 505-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570, 572-599, 601-605, 607-617, 619-622, 624-626, 629-634, 636-645, 647-657, 659-665, 667-668, 670-682]


2021/12/09 21:31:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:31:33 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/09 21:31:33 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/09 21:31:38 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/09 21:31:38 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/09 21:31:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 21:31:38 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/09 21:31:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:31:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:31:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:31:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:31:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 21:31:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 21:31:57 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (491.91s)
=== RUN   TestN1QLRecursion
2021/12/09 21:31:59 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 21:32:00 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 21:32:00 Removing node(s): [] from the cluster
2021/12/09 21:32:05 Rebalance progress: 100
2021/12/09 21:32:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:32:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:32:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:32:27 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:32:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:32:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:32:34 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 21:32:39 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 21:32:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 21:32:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 21:32:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/09 21:32:54 App: TestN1QLRecursion_function1 got deployed
2021/12/09 21:32:54 TestN1QLRecursion_function1 is deployed
2021/12/09 21:32:54 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:32:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 21:32:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:33:00 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 21:33:05 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 21:33:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 21:33:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/09 21:33:15 App: TestN1QLRecursion_function3 got deployed
2021/12/09 21:33:15 TestN1QLRecursion_function3 is deployed
2021/12/09 21:33:15 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:33:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/09 21:33:15 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/09 21:33:15 Success: TestN1QLRecursion_function2 is not deployed
2021/12/09 21:33:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:33:20 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 21:33:25 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 21:33:30 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 21:33:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/09 21:33:35 App: TestN1QLRecursion_function1 got un-deployed
2021/12/09 21:33:35 No eventing-con process running
2021/12/09 21:33:35 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/09 21:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:33:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:33:54 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 21:33:59 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 21:34:04 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 21:34:09 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/09 21:34:09 App: TestN1QLRecursion_function3 got un-deployed
2021/12/09 21:34:09 No eventing-con process running
2021/12/09 21:34:09 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/09 21:34:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:34:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:34:18 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:34:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:34:24 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 21:34:29 Rebalance progress: 100
--- PASS: TestN1QLRecursion (149.79s)
=== RUN   TestN1QLAllowRecursion
2021/12/09 21:34:29 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 21:34:29 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 21:34:29 Removing node(s): [] from the cluster
2021/12/09 21:34:34 Rebalance progress: 100
2021/12/09 21:34:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:34:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:34:58 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:34:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:34:59 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:35:04 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 21:35:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 21:35:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 21:35:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/09 21:35:19 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/09 21:35:19 TestN1QLAllowRecursion_function1 is deployed
2021/12/09 21:35:19 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:35:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/09 21:35:20 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:35:25 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 21:35:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 21:35:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 21:35:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/09 21:35:40 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/09 21:35:40 TestN1QLAllowRecursion_function3 is deployed
2021/12/09 21:35:40 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:35:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/09 21:35:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/09 21:35:46 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 21:35:51 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 21:35:56 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 21:36:01 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/09 21:36:01 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/09 21:36:01 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/09 21:36:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:36:06 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 21:36:11 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 21:36:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 21:36:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/09 21:36:22 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/09 21:36:22 No eventing-con process running
2021/12/09 21:36:22 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/09 21:36:23 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:36:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:36:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:36:40 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 21:36:45 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 21:36:50 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 21:36:55 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/09 21:36:55 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/09 21:36:55 No eventing-con process running
2021/12/09 21:36:55 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/09 21:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:37:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:37:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:37:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:37:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:37:14 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 21:37:19 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 21:37:24 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 21:37:29 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/09 21:37:29 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/09 21:37:29 No eventing-con process running
2021/12/09 21:37:29 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/09 21:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:37:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:37:38 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:37:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:37:45 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 21:37:50 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.69s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/09 21:37:50 Sleeping for some time
2021/12/09 21:38:00 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 21:38:00 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 21:38:00 No eventing-con process running
2021/12/09 21:38:00 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/09 21:38:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:38:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:38:13 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:38:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 21:38:14 Starting bucket ops to source bucket
2021/12/09 21:39:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:39:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 21:39:13 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/09 21:39:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 21:39:13 Starting bucket ops to source bucket
2021/12/09 21:39:15 Starting bucket ops to source bucket
2021/12/09 21:40:20 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/09 21:40:25 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/09 21:40:25 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/09 21:40:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:40:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 21:40:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 21:40:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 21:40:46 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/09 21:40:46 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/09 21:40:46 No eventing-con process running
2021/12/09 21:40:46 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/09 21:40:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:40:57 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/09 21:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:40:59 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:41:09 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/09 21:41:14 Waiting for dst bucket item count to get to: 0 curr count: 3500
2021/12/09 21:41:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.63s)
=== RUN   TestTimerOverWriteSameReference
2021/12/09 21:41:19 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 21:41:19 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 21:41:19 No eventing-con process running
2021/12/09 21:41:19 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/09 21:41:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:41:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:41:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:41:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:41:33 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:41:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/09 21:41:39 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 21:41:44 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 21:41:49 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 21:41:54 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/09 21:41:54 App: TestTimerOverWriteSameReference got deployed
2021/12/09 21:42:04 Starting bucket ops to source bucket
2021/12/09 21:43:05 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/09 21:43:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:43:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:43:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 21:43:25 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 21:43:30 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 21:43:35 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/09 21:43:35 App: TestTimerOverWriteSameReference got un-deployed
2021/12/09 21:43:35 No eventing-con process running
2021/12/09 21:43:35 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/09 21:43:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:43:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:43:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:43:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:43:59 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/09 21:43:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (164.67s)
=== RUN   TestCancelTimerBucketop
2021/12/09 21:44:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 21:44:04 App: TestCancelTimerBucketop got un-deployed
2021/12/09 21:44:04 No eventing-con process running
2021/12/09 21:44:04 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/09 21:44:06 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:44:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:44:13 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:44:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:44:18 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:44:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/09 21:44:24 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 21:44:29 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 21:44:34 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 21:44:39 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/09 21:44:39 App: TestCancelTimerBucketop got deployed
2021/12/09 21:44:39 Starting bucket ops to source bucket
2021/12/09 21:44:46 Starting bucket ops to source bucket
2021/12/09 21:45:48 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 21:45:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:45:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:45:53 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 21:45:58 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 21:46:03 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 21:46:08 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/09 21:46:08 App: TestCancelTimerBucketop got un-deployed
2021/12/09 21:46:08 No eventing-con process running
2021/12/09 21:46:08 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/09 21:46:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:46:20 Waiting for dst bucket item count to get to: 0 curr count: 667
2021/12/09 21:46:25 Waiting for dst bucket item count to get to: 0 curr count: 667
2021/12/09 21:46:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:46:27 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:46:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:46:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.16s)
=== RUN   TestJSExpiryDate
2021/12/09 21:46:42 Starting bucket ops to source bucket
2021/12/09 21:46:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 21:46:55 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/09 21:46:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/09 21:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:47:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:47:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:47:32 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 21:47:37 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 21:47:42 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 21:47:47 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/09 21:47:47 App: TestJSExpiryDate got un-deployed
2021/12/09 21:47:47 No eventing-con process running
2021/12/09 21:47:47 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/09 21:47:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:47:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.75s)
=== RUN   TestBinaryDoc
2021/12/09 21:48:06 Starting bucket ops to source bucket
2021/12/09 21:48:06 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/09 21:48:07 Starting bucket ops to source bucket
2021/12/09 21:48:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:48:07 Deleting source bucket docs now
2021/12/09 21:48:07 Starting bucket ops to source bucket
2021/12/09 21:48:44 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/09 21:48:49 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 21:48:54 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/09 21:48:54 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/09 21:48:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:48:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:48:59 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 21:49:04 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 21:49:09 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 21:49:14 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/09 21:49:14 App: TestBinaryDoc got un-deployed
2021/12/09 21:49:14 No eventing-con process running
2021/12/09 21:49:15 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/09 21:49:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:49:26 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/09 21:49:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:49:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:49:38 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/09 21:49:43 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/09 21:49:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.17s)
=== RUN   TestConstantBindings
2021/12/09 21:49:43 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:49:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/09 21:49:44 Starting bucket ops to source bucket
2021/12/09 21:49:50 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 21:49:55 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 21:50:00 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/09 21:50:05 Waiting for dst bucket item count to get to: 6000 curr count: 1903
2021/12/09 21:50:10 Waiting for dst bucket item count to get to: 6000 curr count: 4903
2021/12/09 21:50:10 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/09 21:50:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:50:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:50:15 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 21:50:20 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 21:50:25 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 21:50:30 Waiting for app: TestConstantBindings to get un-deployed
2021/12/09 21:50:30 App: TestConstantBindings got un-deployed
2021/12/09 21:50:30 No eventing-con process running
2021/12/09 21:50:30 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/09 21:50:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:50:42 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:50:47 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:50:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:50:59 Waiting for dst bucket item count to get to: 0 curr count: 5418
2021/12/09 21:51:04 Waiting for dst bucket item count to get to: 0 curr count: 2418
2021/12/09 21:51:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.63s)
=== RUN   TestCodeInsight
2021/12/09 21:51:09 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 21:51:09 App: TestCodeInsight got un-deployed
2021/12/09 21:51:09 No eventing-con process running
2021/12/09 21:51:09 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/09 21:51:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:51:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:51:22 Starting bucket ops to source bucket
2021/12/09 21:51:23 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:51:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/09 21:51:29 Waiting for app: TestCodeInsight to get deployed
2021/12/09 21:51:34 Waiting for app: TestCodeInsight to get deployed
2021/12/09 21:51:39 Waiting for app: TestCodeInsight to get deployed
2021/12/09 21:51:44 Waiting for app: TestCodeInsight to get deployed
2021/12/09 21:51:44 App: TestCodeInsight got deployed
2021/12/09 21:51:49 Waiting for dst bucket item count to get to: 1000 curr count: 912
2021/12/09 21:51:49 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 21:51:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:51:54 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 21:51:59 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 21:52:04 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 21:52:09 Waiting for app: TestCodeInsight to get un-deployed
2021/12/09 21:52:09 App: TestCodeInsight got un-deployed
2021/12/09 21:52:09 No eventing-con process running
2021/12/09 21:52:09 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/09 21:52:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:52:21 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:52:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:52:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:52:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:52:38 Waiting for dst bucket item count to get to: 0 curr count: 671
2021/12/09 21:52:43 Waiting for dst bucket item count to get to: 0 curr count: 171
2021/12/09 21:52:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.30s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13013.163s

Functional tests took 3 hours, 36 minutes and 57 seconds.

N1ql tests

2021/12/09 21:52:53 Started n1ql
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/09 21:53:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/09 21:53:03 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/09 21:53:03 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 21:53:04 Removing node(s): [] from the cluster
2021/12/09 21:53:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/09 21:53:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/09 21:53:14 App: TestRecursiveMutationN1QL got un-deployed
2021/12/09 21:53:14 No eventing-con process running
2021/12/09 21:53:14 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/09 21:53:16 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:53:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:53:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:53:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:53:27 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:53:27 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.61s)
=== RUN   TestFlexReset1
2021/12/09 21:53:32 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 21:53:32 App: TestFlexReset1 got un-deployed
2021/12/09 21:53:32 No eventing-con process running
2021/12/09 21:53:32 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/09 21:53:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:53:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:53:46 Starting bucket ops to source bucket
2021/12/09 21:53:47 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:53:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/09 21:53:53 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 21:53:58 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 21:54:03 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 21:54:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/09 21:54:08 App: TestFlexReset1 got deployed
2021/12/09 21:54:13 Waiting for dst bucket item count to get to: 1000 curr count: 28
2021/12/09 21:54:13 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 21:54:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:54:18 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 21:54:23 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 21:54:28 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 21:54:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/09 21:54:33 App: TestFlexReset1 got un-deployed
2021/12/09 21:54:33 No eventing-con process running
2021/12/09 21:54:33 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/09 21:54:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:54:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:54:46 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:54:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:54:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.16s)
=== RUN   TestFlexReset2
2021/12/09 21:55:01 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 21:55:01 App: TestFlexReset2 got un-deployed
2021/12/09 21:55:01 No eventing-con process running
2021/12/09 21:55:01 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/09 21:55:03 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:55:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:55:10 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:55:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:55:15 Starting bucket ops to source bucket
2021/12/09 21:55:16 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:55:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/09 21:55:22 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 21:55:27 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 21:55:32 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 21:55:37 Waiting for app: TestFlexReset2 to get deployed
2021/12/09 21:55:37 App: TestFlexReset2 got deployed
2021/12/09 21:55:42 Waiting for dst bucket item count to get to: 1000 curr count: 106
2021/12/09 21:55:42 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/09 21:55:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:55:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:55:47 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 21:55:52 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 21:55:57 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 21:56:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/09 21:56:02 App: TestFlexReset2 got un-deployed
2021/12/09 21:56:02 No eventing-con process running
2021/12/09 21:56:02 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/09 21:56:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:56:14 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/09 21:56:19 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 21:56:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:56:21 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:56:31 Waiting for dst bucket item count to get to: 0 curr count: 903
2021/12/09 21:56:36 Waiting for dst bucket item count to get to: 0 curr count: 903
2021/12/09 21:56:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.23s)
=== RUN   TestN1QLLabelledBreak
2021/12/09 21:56:41 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 21:56:41 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 21:56:41 No eventing-con process running
2021/12/09 21:56:41 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/09 21:56:42 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:56:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:56:54 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:56:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 21:57:00 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 21:57:05 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 21:57:10 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 21:57:15 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/09 21:57:15 App: TestN1QLLabelledBreak got deployed
2021/12/09 21:57:15 Starting bucket ops to source bucket
2021/12/09 21:57:25 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 21:57:30 Waiting for dst bucket item count to get to: 10000 curr count: 375
2021/12/09 21:57:35 Waiting for dst bucket item count to get to: 10000 curr count: 375
2021/12/09 21:57:40 Waiting for dst bucket item count to get to: 10000 curr count: 4076
2021/12/09 21:57:45 Waiting for dst bucket item count to get to: 10000 curr count: 4076
2021/12/09 21:57:50 Waiting for dst bucket item count to get to: 10000 curr count: 8152
2021/12/09 21:57:55 Waiting for dst bucket item count to get to: 10000 curr count: 8152
2021/12/09 21:57:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 21:57:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:57:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 21:58:00 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 21:58:05 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 21:58:10 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 21:58:15 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/09 21:58:15 App: TestN1QLLabelledBreak got un-deployed
2021/12/09 21:58:15 No eventing-con process running
2021/12/09 21:58:15 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/09 21:58:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:58:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 21:58:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:58:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:58:39 Waiting for dst bucket item count to get to: 0 curr count: 8708
2021/12/09 21:58:44 Waiting for dst bucket item count to get to: 0 curr count: 3708
2021/12/09 21:58:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (127.85s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/09 21:58:49 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 21:58:49 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 21:58:49 No eventing-con process running
2021/12/09 21:58:49 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/09 21:58:50 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:58:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:58:57 Context: Bucket flush, response code: 200 dump: 
2021/12/09 21:59:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 21:59:02 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/09 21:59:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/09 21:59:08 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 21:59:13 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 21:59:18 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 21:59:23 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/09 21:59:23 App: TestN1QLUnlabelledBreak got deployed
2021/12/09 21:59:23 Starting bucket ops to source bucket
2021/12/09 21:59:33 Waiting for dst bucket item count to get to: 10000 curr count: 439
2021/12/09 21:59:38 Waiting for dst bucket item count to get to: 10000 curr count: 439
2021/12/09 21:59:43 Waiting for dst bucket item count to get to: 10000 curr count: 1348
2021/12/09 21:59:48 Waiting for dst bucket item count to get to: 10000 curr count: 4118
2021/12/09 21:59:53 Waiting for dst bucket item count to get to: 10000 curr count: 7014
2021/12/09 21:59:58 Waiting for dst bucket item count to get to: 10000 curr count: 8805
2021/12/09 21:59:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 21:59:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 21:59:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:00:03 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 22:00:08 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 22:00:13 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 22:00:18 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/09 22:00:18 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/09 22:00:18 No eventing-con process running
2021/12/09 22:00:18 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/09 22:00:20 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:00:30 Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/09 22:00:35 Waiting for dst bucket item count to get to: 0 curr count: 3177
2021/12/09 22:00:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:00:37 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:00:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 22:00:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (123.16s)
=== RUN   TestN1QLThrowStatement
2021/12/09 22:00:52 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 22:00:52 App: TestN1QLThrowStatement got un-deployed
2021/12/09 22:00:52 No eventing-con process running
2021/12/09 22:00:52 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/09 22:00:53 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:01:00 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:01:05 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:01:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/09 22:01:11 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 22:01:16 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 22:01:21 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 22:01:26 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/09 22:01:26 App: TestN1QLThrowStatement got deployed
2021/12/09 22:01:26 Starting bucket ops to source bucket
2021/12/09 22:01:36 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/09 22:01:41 Waiting for dst bucket item count to get to: 10000 curr count: 239
2021/12/09 22:01:46 Waiting for dst bucket item count to get to: 10000 curr count: 239
2021/12/09 22:01:51 Waiting for dst bucket item count to get to: 10000 curr count: 3638
2021/12/09 22:01:56 Waiting for dst bucket item count to get to: 10000 curr count: 3638
2021/12/09 22:02:01 Waiting for dst bucket item count to get to: 10000 curr count: 7653
2021/12/09 22:02:06 Waiting for dst bucket item count to get to: 10000 curr count: 7653
2021/12/09 22:02:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/09 22:02:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:02:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:02:11 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 22:02:16 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 22:02:21 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 22:02:26 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/09 22:02:26 App: TestN1QLThrowStatement got un-deployed
2021/12/09 22:02:26 No eventing-con process running
2021/12/09 22:02:26 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/09 22:02:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:02:38 Waiting for dst bucket item count to get to: 0 curr count: 3015
2021/12/09 22:02:43 Waiting for dst bucket item count to get to: 0 curr count: 515
2021/12/09 22:02:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:02:45 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:02:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/09 22:02:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.91s)
=== RUN   TestN1QLNestedForLoop
2021/12/09 22:03:00 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 22:03:00 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 22:03:00 No eventing-con process running
2021/12/09 22:03:00 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/09 22:03:01 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:03:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:03:08 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:03:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:03:13 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:03:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/09 22:03:19 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 22:03:24 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 22:03:29 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 22:03:34 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/09 22:03:34 App: TestN1QLNestedForLoop got deployed
2021/12/09 22:03:34 Starting bucket ops to source bucket
2021/12/09 22:03:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 22:03:49 Waiting for dst bucket item count to get to: 5000 curr count: 95
2021/12/09 22:03:54 Waiting for dst bucket item count to get to: 5000 curr count: 318
2021/12/09 22:03:59 Waiting for dst bucket item count to get to: 5000 curr count: 1392
2021/12/09 22:04:04 Waiting for dst bucket item count to get to: 5000 curr count: 2596
2021/12/09 22:04:09 Waiting for dst bucket item count to get to: 5000 curr count: 3803
2021/12/09 22:04:14 Waiting for dst bucket item count to get to: 5000 curr count: 4876
2021/12/09 22:04:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 22:04:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:04:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:04:19 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 22:04:24 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 22:04:29 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 22:04:34 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/09 22:04:34 App: TestN1QLNestedForLoop got un-deployed
2021/12/09 22:04:34 No eventing-con process running
2021/12/09 22:04:34 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/09 22:04:36 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:04:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 22:04:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:04:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:04:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/09 22:04:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.90s)
=== RUN   TestN1QLAutoClose
2021/12/09 22:05:03 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 22:05:03 App: TestN1QLAutoClose got un-deployed
2021/12/09 22:05:03 No eventing-con process running
2021/12/09 22:05:03 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/09 22:05:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:05:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:05:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:05:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:05:16 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:05:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/09 22:05:22 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 22:05:27 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 22:05:32 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 22:05:37 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/09 22:05:37 App: TestN1QLAutoClose got deployed
2021/12/09 22:05:37 Starting bucket ops to source bucket
2021/12/09 22:05:43 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/09 22:05:48 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/09 22:05:53 Waiting for dst bucket item count to get to: 500 curr count: 1
2021/12/09 22:05:58 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/09 22:06:03 Waiting for dst bucket item count to get to: 500 curr count: 24
2021/12/09 22:06:08 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/09 22:06:13 Waiting for dst bucket item count to get to: 500 curr count: 45
2021/12/09 22:06:18 Waiting for dst bucket item count to get to: 500 curr count: 66
2021/12/09 22:06:23 Waiting for dst bucket item count to get to: 500 curr count: 66
2021/12/09 22:06:28 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/09 22:06:33 Waiting for dst bucket item count to get to: 500 curr count: 99
2021/12/09 22:06:38 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/09 22:06:43 Waiting for dst bucket item count to get to: 500 curr count: 121
2021/12/09 22:06:48 Waiting for dst bucket item count to get to: 500 curr count: 121
2021/12/09 22:06:53 Waiting for dst bucket item count to get to: 500 curr count: 133
2021/12/09 22:06:58 Waiting for dst bucket item count to get to: 500 curr count: 143
2021/12/09 22:07:03 Waiting for dst bucket item count to get to: 500 curr count: 154
2021/12/09 22:07:08 Waiting for dst bucket item count to get to: 500 curr count: 164
2021/12/09 22:07:13 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/09 22:07:18 Waiting for dst bucket item count to get to: 500 curr count: 188
2021/12/09 22:07:23 Waiting for dst bucket item count to get to: 500 curr count: 198
2021/12/09 22:07:28 Waiting for dst bucket item count to get to: 500 curr count: 208
2021/12/09 22:07:33 Waiting for dst bucket item count to get to: 500 curr count: 220
2021/12/09 22:07:38 Waiting for dst bucket item count to get to: 500 curr count: 229
2021/12/09 22:07:43 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/09 22:07:48 Waiting for dst bucket item count to get to: 500 curr count: 250
2021/12/09 22:07:53 Waiting for dst bucket item count to get to: 500 curr count: 262
2021/12/09 22:07:58 Waiting for dst bucket item count to get to: 500 curr count: 272
2021/12/09 22:08:03 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/09 22:08:08 Waiting for dst bucket item count to get to: 500 curr count: 306
2021/12/09 22:08:13 Waiting for dst bucket item count to get to: 500 curr count: 306
2021/12/09 22:08:18 Waiting for dst bucket item count to get to: 500 curr count: 327
2021/12/09 22:08:23 Waiting for dst bucket item count to get to: 500 curr count: 327
2021/12/09 22:08:28 Waiting for dst bucket item count to get to: 500 curr count: 348
2021/12/09 22:08:33 Waiting for dst bucket item count to get to: 500 curr count: 348
2021/12/09 22:08:38 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/09 22:08:43 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/09 22:08:48 Waiting for dst bucket item count to get to: 500 curr count: 389
2021/12/09 22:08:53 Waiting for dst bucket item count to get to: 500 curr count: 389
2021/12/09 22:08:58 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/09 22:09:03 Waiting for dst bucket item count to get to: 500 curr count: 410
2021/12/09 22:09:08 Waiting for dst bucket item count to get to: 500 curr count: 431
2021/12/09 22:09:13 Waiting for dst bucket item count to get to: 500 curr count: 431
2021/12/09 22:09:18 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/09 22:09:23 Waiting for dst bucket item count to get to: 500 curr count: 453
2021/12/09 22:09:28 Waiting for dst bucket item count to get to: 500 curr count: 477
2021/12/09 22:09:33 Waiting for dst bucket item count to get to: 500 curr count: 477
2021/12/09 22:09:38 Waiting for dst bucket item count to get to: 500 curr count: 496
2021/12/09 22:09:43 Waiting for dst bucket item count to get to: 500 curr count: 496
2021/12/09 22:09:43 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/09 22:09:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:09:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:09:49 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 22:09:54 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 22:09:59 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 22:10:04 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/09 22:10:04 App: TestN1QLAutoClose got un-deployed
2021/12/09 22:10:04 No eventing-con process running
2021/12/09 22:10:04 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/09 22:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:10:15 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 22:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:10:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/09 22:10:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (329.64s)
=== RUN   TestN1QLPosParams
2021/12/09 22:10:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 22:10:32 App: TestN1QLPosParams got un-deployed
2021/12/09 22:10:32 No eventing-con process running
2021/12/09 22:10:32 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/09 22:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:10:41 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:10:46 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:10:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/09 22:10:52 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 22:10:57 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 22:11:02 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 22:11:07 Waiting for app: TestN1QLPosParams to get deployed
2021/12/09 22:11:07 App: TestN1QLPosParams got deployed
2021/12/09 22:11:07 Starting bucket ops to source bucket
2021/12/09 22:11:16 Waiting for dst bucket item count to get to: 5000 curr count: 427
2021/12/09 22:11:21 Waiting for dst bucket item count to get to: 5000 curr count: 427
2021/12/09 22:11:26 Waiting for dst bucket item count to get to: 5000 curr count: 427
2021/12/09 22:11:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 22:11:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:11:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:11:32 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 22:11:37 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 22:11:42 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 22:11:47 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/09 22:11:47 App: TestN1QLPosParams got un-deployed
2021/12/09 22:11:47 No eventing-con process running
2021/12/09 22:11:47 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/09 22:11:48 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:11:58 Waiting for dst bucket item count to get to: 0 curr count: 4287
2021/12/09 22:12:03 Waiting for dst bucket item count to get to: 0 curr count: 1787
2021/12/09 22:12:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:12:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 22:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.07s)
=== RUN   TestN1QLExhaustConnPool
2021/12/09 22:12:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 22:12:20 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 22:12:20 No eventing-con process running
2021/12/09 22:12:20 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/09 22:12:22 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:12:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:12:34 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:12:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/09 22:12:40 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 22:12:45 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 22:12:50 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 22:12:55 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/09 22:12:55 App: TestN1QLExhaustConnPool got deployed
2021/12/09 22:12:55 Starting bucket ops to source bucket
2021/12/09 22:13:00 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/09 22:13:05 Waiting for dst bucket item count to get to: 100 curr count: 31
2021/12/09 22:13:10 Waiting for dst bucket item count to get to: 100 curr count: 81
2021/12/09 22:13:10 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/09 22:13:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:13:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:13:15 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 22:13:20 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 22:13:25 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 22:13:30 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/09 22:13:30 App: TestN1QLExhaustConnPool got un-deployed
2021/12/09 22:13:30 No eventing-con process running
2021/12/09 22:13:30 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/09 22:13:32 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:13:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 22:13:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 22:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:13:49 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:13:59 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/09 22:14:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/09 22:14:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (108.69s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/09 22:14:09 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 22:14:09 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 22:14:09 No eventing-con process running
2021/12/09 22:14:09 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/09 22:14:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:14:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:14:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:14:22 Starting bucket ops to source bucket
2021/12/09 22:14:40 Starting bucket ops to source bucket
2021/12/09 22:14:57 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 15254
2021/12/09 22:14:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/09 22:15:03 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 22:15:08 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 22:15:13 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 22:15:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/09 22:15:18 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/09 22:15:18 Starting bucket ops to source bucket
2021/12/09 22:15:23 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:28 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:33 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:38 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:43 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:48 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:53 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:15:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:16:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:16:08 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:16:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/09 22:16:13 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/09 22:16:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:16:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:16:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 22:16:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 22:16:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 22:16:33 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/09 22:16:33 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/09 22:16:33 No eventing-con process running
2021/12/09 22:16:33 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/09 22:16:35 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:16:45 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/09 22:16:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:16:47 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:16:57 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/09 22:16:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (173.07s)
=== RUN   TestN1QLQueryConsistency
2021/12/09 22:17:02 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 22:17:02 App: n1qlquery_consistency got un-deployed
2021/12/09 22:17:02 No eventing-con process running
2021/12/09 22:17:02 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/09 22:17:04 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:17:11 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:17:16 Starting bucket ops to source bucket
2021/12/09 22:17:19 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:17:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/09 22:17:25 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 22:17:30 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 22:17:35 Waiting for app: n1qlquery_consistency to get deployed
2021/12/09 22:17:35 App: n1qlquery_consistency got deployed
2021/12/09 22:17:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 22:17:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/09 22:17:50 Waiting for dst bucket item count to get to: 5000 curr count: 4004
2021/12/09 22:17:55 Waiting for dst bucket item count to get to: 5000 curr count: 4004
2021/12/09 22:17:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/09 22:17:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/09 22:17:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:18:00 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 22:18:05 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 22:18:10 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 22:18:15 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/09 22:18:15 App: n1qlquery_consistency got un-deployed
2021/12/09 22:18:15 No eventing-con process running
2021/12/09 22:18:15 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/09 22:18:17 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:18:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/09 22:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:18:28 Context: Bucket flush, response code: 200 dump: 
2021/12/09 22:18:38 Waiting for dst bucket item count to get to: 0 curr count: 4388
2021/12/09 22:18:43 Waiting for dst bucket item count to get to: 0 curr count: 1888
2021/12/09 22:18:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.47s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1546.557s

N1ql tests took 0 hours, 25 minutes and 51 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/09 22:20:19 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/09 22:20:36 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/09 22:20:41 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 22:20:41 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 22:20:41 No eventing-con process running
2021/12/09 22:20:41 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/09 22:20:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:20:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:20:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:20:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:20:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:21:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:21:00 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:21:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 22:21:06 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 22:21:11 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 22:21:16 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 22:21:21 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/09 22:21:21 App: TestDuplicateWithEventingRebIn got deployed
2021/12/09 22:21:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:21:21 Starting bucket ops to source bucket
2021/12/09 22:21:36 Removing node(s): [] from the cluster
2021/12/09 22:21:41 Rebalance progress: 0
2021/12/09 22:21:46 Rebalance progress: 0
2021/12/09 22:21:51 Rebalance progress: 25
2021/12/09 22:21:56 Rebalance progress: 25
2021/12/09 22:22:01 Rebalance progress: 25
2021/12/09 22:22:06 Rebalance progress: 98.61111111111111
2021/12/09 22:22:11 Rebalance progress: 98.61111111111111
2021/12/09 22:22:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:22:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:22:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 22:22:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:22:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:22:21 Rebalance progress: 25
2021/12/09 22:22:26 Rebalance progress: 25
2021/12/09 22:22:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:22:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:22:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 22:22:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:22:31 Waiting for src bucket item count to get to: 200000 curr count: 95097
2021/12/09 22:22:36 Waiting for src bucket item count to get to: 200000 curr count: 112093
2021/12/09 22:22:41 Waiting for src bucket item count to get to: 200000 curr count: 112093
2021/12/09 22:22:46 Waiting for src bucket item count to get to: 200000 curr count: 128733
2021/12/09 22:22:51 Waiting for src bucket item count to get to: 200000 curr count: 128733
2021/12/09 22:22:56 Waiting for src bucket item count to get to: 200000 curr count: 145633
2021/12/09 22:23:01 Waiting for src bucket item count to get to: 200000 curr count: 145633
2021/12/09 22:23:06 Waiting for src bucket item count to get to: 200000 curr count: 162417
2021/12/09 22:23:11 Waiting for src bucket item count to get to: 200000 curr count: 162417
2021/12/09 22:23:16 Waiting for src bucket item count to get to: 200000 curr count: 178898
2021/12/09 22:23:21 Waiting for src bucket item count to get to: 200000 curr count: 178898
2021/12/09 22:23:26 Waiting for src bucket item count to get to: 200000 curr count: 195531
2021/12/09 22:23:31 Waiting for src bucket item count to get to: 200000 curr count: 195531
2021/12/09 22:23:41 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:23:46 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:23:51 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:23:56 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:01 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:06 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:11 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:16 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:21 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:26 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/09 22:24:26 Post SET ops. Expected item count: 200000 got 200001
2021/12/09 22:24:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:24:31 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 22:24:36 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 22:24:41 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 22:24:46 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/09 22:24:46 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/09 22:24:46 No eventing-con process running
2021/12/09 22:24:46 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/09 22:24:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:24:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:25:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:25:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (278.44s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/09 22:25:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:25:10 Removing node(s): [] from the cluster
2021/12/09 22:25:15 Rebalance progress: 100
2021/12/09 22:25:20 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 22:25:20 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 22:25:20 No eventing-con process running
2021/12/09 22:25:20 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/09 22:25:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:25:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:25:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:25:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:25:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:25:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:25:38 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:25:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 22:25:45 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 22:25:50 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 22:25:55 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 22:26:00 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/09 22:26:00 App: TestDuplicateWithEventingRebOut got deployed
2021/12/09 22:26:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:26:00 Starting bucket ops to source bucket
2021/12/09 22:26:05 Rebalance progress: 25
2021/12/09 22:26:10 Rebalance progress: 25
2021/12/09 22:26:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:26:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:26:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:26:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 22:26:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:26:15 Waiting for src bucket item count to get to: 200000 curr count: 15720
2021/12/09 22:26:20 Waiting for src bucket item count to get to: 200000 curr count: 15720
2021/12/09 22:26:25 Waiting for src bucket item count to get to: 200000 curr count: 32723
2021/12/09 22:26:30 Waiting for src bucket item count to get to: 200000 curr count: 32723
2021/12/09 22:26:35 Waiting for src bucket item count to get to: 200000 curr count: 49816
2021/12/09 22:26:40 Waiting for src bucket item count to get to: 200000 curr count: 66853
2021/12/09 22:26:45 Waiting for src bucket item count to get to: 200000 curr count: 66853
2021/12/09 22:26:50 Waiting for src bucket item count to get to: 200000 curr count: 66853
2021/12/09 22:26:55 Waiting for src bucket item count to get to: 200000 curr count: 83399
2021/12/09 22:27:00 Waiting for src bucket item count to get to: 200000 curr count: 83399
2021/12/09 22:27:05 Waiting for src bucket item count to get to: 200000 curr count: 99833
2021/12/09 22:27:10 Waiting for src bucket item count to get to: 200000 curr count: 99833
2021/12/09 22:27:15 Waiting for src bucket item count to get to: 200000 curr count: 116699
2021/12/09 22:27:20 Waiting for src bucket item count to get to: 200000 curr count: 116699
2021/12/09 22:27:25 Waiting for src bucket item count to get to: 200000 curr count: 133777
2021/12/09 22:27:30 Waiting for src bucket item count to get to: 200000 curr count: 133777
2021/12/09 22:27:35 Waiting for src bucket item count to get to: 200000 curr count: 150648
2021/12/09 22:27:40 Waiting for src bucket item count to get to: 200000 curr count: 150648
2021/12/09 22:27:45 Waiting for src bucket item count to get to: 200000 curr count: 167761
2021/12/09 22:27:50 Waiting for src bucket item count to get to: 200000 curr count: 167761
2021/12/09 22:27:55 Waiting for src bucket item count to get to: 200000 curr count: 184168
2021/12/09 22:28:00 Waiting for src bucket item count to get to: 200000 curr count: 184168
2021/12/09 22:28:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 22:28:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 22:28:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:28:10 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 22:28:15 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 22:28:20 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 22:28:25 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/09 22:28:25 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/09 22:28:25 No eventing-con process running
2021/12/09 22:28:25 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/09 22:28:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:28:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:28:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:28:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:28:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:28:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:28:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (223.74s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/09 22:28:58 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 22:28:58 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 22:28:58 No eventing-con process running
2021/12/09 22:28:58 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/09 22:29:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:29:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:29:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:29:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:29:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:29:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:29:17 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:29:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 22:29:23 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 22:29:28 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 22:29:33 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 22:29:38 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/09 22:29:38 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/09 22:29:38 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:29:38 Starting bucket ops to source bucket
2021/12/09 22:29:38 Removing node(s): [] from the cluster
2021/12/09 22:29:43 Rebalance progress: 25
2021/12/09 22:29:48 Rebalance progress: 25
2021/12/09 22:29:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:29:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:29:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:29:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/09 22:29:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:29:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:30:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:30:05 Rebalance progress: 16.66666666666667
2021/12/09 22:30:10 Rebalance progress: 16.66666666666667
2021/12/09 22:30:15 Rebalance progress: 16.66666666666667
2021/12/09 22:30:20 Rebalance progress: 29.46317103620475
2021/12/09 22:30:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:30:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:30:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:30:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 22:30:31 Rebalance progress: 16.66666666666667
2021/12/09 22:30:36 Rebalance progress: 16.66666666666667
2021/12/09 22:30:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:30:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:30:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:30:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:30:41 Waiting for src bucket item count to get to: 200000 curr count: 77097
2021/12/09 22:30:46 Waiting for src bucket item count to get to: 200000 curr count: 93537
2021/12/09 22:30:51 Waiting for src bucket item count to get to: 200000 curr count: 93537
2021/12/09 22:30:56 Waiting for src bucket item count to get to: 200000 curr count: 109771
2021/12/09 22:31:01 Waiting for src bucket item count to get to: 200000 curr count: 109771
2021/12/09 22:31:06 Waiting for src bucket item count to get to: 200000 curr count: 126169
2021/12/09 22:31:11 Waiting for src bucket item count to get to: 200000 curr count: 126169
2021/12/09 22:31:16 Waiting for src bucket item count to get to: 200000 curr count: 142589
2021/12/09 22:31:21 Waiting for src bucket item count to get to: 200000 curr count: 142589
2021/12/09 22:31:26 Waiting for src bucket item count to get to: 200000 curr count: 159233
2021/12/09 22:31:31 Waiting for src bucket item count to get to: 200000 curr count: 159233
2021/12/09 22:31:36 Waiting for src bucket item count to get to: 200000 curr count: 175335
2021/12/09 22:31:41 Waiting for src bucket item count to get to: 200000 curr count: 175335
2021/12/09 22:31:46 Waiting for src bucket item count to get to: 200000 curr count: 191889
2021/12/09 22:31:51 Waiting for src bucket item count to get to: 200000 curr count: 191889
2021/12/09 22:31:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 22:31:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 22:31:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:32:01 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 22:32:06 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 22:32:11 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 22:32:16 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/09 22:32:16 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/09 22:32:16 No eventing-con process running
2021/12/09 22:32:16 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/09 22:32:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:32:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:32:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:32:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:32:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:32:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:32:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:32:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.90s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/09 22:32:49 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 22:32:49 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 22:32:49 No eventing-con process running
2021/12/09 22:32:49 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/09 22:32:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:32:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:32:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:33:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:33:08 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:33:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/09 22:33:14 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 22:33:19 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 22:33:24 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 22:33:29 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/09 22:33:29 App: TestDuplicateWithKVRebIn got deployed
2021/12/09 22:33:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 22:33:29 Starting bucket ops to source bucket
2021/12/09 22:33:29 Removing node(s): [] from the cluster
2021/12/09 22:33:34 Rebalance progress: 16.66666666666667
2021/12/09 22:33:39 Rebalance progress: 16.66666666666667
2021/12/09 22:33:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:33:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:33:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:33:49 Rebalance progress: 16.66666666666667
2021/12/09 22:33:54 Rebalance progress: 16.66666666666667
2021/12/09 22:33:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:33:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:33:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:33:59 Waiting for src bucket item count to get to: 200000 curr count: 31875
2021/12/09 22:34:04 Waiting for src bucket item count to get to: 200000 curr count: 48049
2021/12/09 22:34:09 Waiting for src bucket item count to get to: 200000 curr count: 48049
2021/12/09 22:34:14 Waiting for src bucket item count to get to: 200000 curr count: 64701
2021/12/09 22:34:19 Waiting for src bucket item count to get to: 200000 curr count: 64701
2021/12/09 22:34:24 Waiting for src bucket item count to get to: 200000 curr count: 80487
2021/12/09 22:34:29 Waiting for src bucket item count to get to: 200000 curr count: 80487
2021/12/09 22:34:34 Waiting for src bucket item count to get to: 200000 curr count: 96634
2021/12/09 22:34:39 Waiting for src bucket item count to get to: 200000 curr count: 96634
2021/12/09 22:34:44 Waiting for src bucket item count to get to: 200000 curr count: 112883
2021/12/09 22:34:49 Waiting for src bucket item count to get to: 200000 curr count: 112883
2021/12/09 22:34:54 Waiting for src bucket item count to get to: 200000 curr count: 112883
2021/12/09 22:34:59 Waiting for src bucket item count to get to: 200000 curr count: 129143
2021/12/09 22:35:04 Waiting for src bucket item count to get to: 200000 curr count: 145457
2021/12/09 22:35:09 Waiting for src bucket item count to get to: 200000 curr count: 145457
2021/12/09 22:35:15 Waiting for src bucket item count to get to: 200000 curr count: 162053
2021/12/09 22:35:20 Waiting for src bucket item count to get to: 200000 curr count: 162053
2021/12/09 22:35:25 Waiting for src bucket item count to get to: 200000 curr count: 178084
2021/12/09 22:35:30 Waiting for src bucket item count to get to: 200000 curr count: 178084
2021/12/09 22:35:35 Waiting for src bucket item count to get to: 200000 curr count: 193882
2021/12/09 22:35:40 Waiting for src bucket item count to get to: 200000 curr count: 193882
2021/12/09 22:35:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 22:35:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 22:35:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:35:50 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 22:35:55 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 22:36:00 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 22:36:05 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/09 22:36:05 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/09 22:36:05 No eventing-con process running
2021/12/09 22:36:05 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/09 22:36:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:36:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:36:16 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:36:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:36:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:36:28 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:36:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (228.94s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/09 22:36:33 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 22:36:34 Removing node(s): [] from the cluster
2021/12/09 22:36:39 Rebalance progress: 100
2021/12/09 22:36:44 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 22:36:44 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 22:36:44 No eventing-con process running
2021/12/09 22:36:44 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/09 22:36:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:36:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:36:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:36:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:36:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:36:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:37:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:37:02 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:37:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/09 22:37:08 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 22:37:13 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 22:37:18 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 22:37:23 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/09 22:37:23 App: TestDuplicateWithKVRebOut got deployed
2021/12/09 22:37:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:37:23 Starting bucket ops to source bucket
2021/12/09 22:37:28 Rebalance progress: 16.66666666666667
2021/12/09 22:37:33 Rebalance progress: 16.66666666666667
2021/12/09 22:37:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:37:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:37:38 Waiting for src bucket item count to get to: 200000 curr count: 8986
2021/12/09 22:37:43 Waiting for src bucket item count to get to: 200000 curr count: 25239
2021/12/09 22:37:48 Waiting for src bucket item count to get to: 200000 curr count: 25239
2021/12/09 22:37:53 Waiting for src bucket item count to get to: 200000 curr count: 41783
2021/12/09 22:37:58 Waiting for src bucket item count to get to: 200000 curr count: 41783
2021/12/09 22:38:03 Waiting for src bucket item count to get to: 200000 curr count: 58342
2021/12/09 22:38:08 Waiting for src bucket item count to get to: 200000 curr count: 58342
2021/12/09 22:38:13 Waiting for src bucket item count to get to: 200000 curr count: 58342
2021/12/09 22:38:18 Waiting for src bucket item count to get to: 200000 curr count: 74615
2021/12/09 22:38:23 Waiting for src bucket item count to get to: 200000 curr count: 74615
2021/12/09 22:38:28 Waiting for src bucket item count to get to: 200000 curr count: 90967
2021/12/09 22:38:34 Waiting for src bucket item count to get to: 200000 curr count: 90967
2021/12/09 22:38:39 Waiting for src bucket item count to get to: 200000 curr count: 107784
2021/12/09 22:38:44 Waiting for src bucket item count to get to: 200000 curr count: 107784
2021/12/09 22:38:49 Waiting for src bucket item count to get to: 200000 curr count: 124350
2021/12/09 22:38:54 Waiting for src bucket item count to get to: 200000 curr count: 124350
2021/12/09 22:38:59 Waiting for src bucket item count to get to: 200000 curr count: 141029
2021/12/09 22:39:04 Waiting for src bucket item count to get to: 200000 curr count: 141029
2021/12/09 22:39:09 Waiting for src bucket item count to get to: 200000 curr count: 157587
2021/12/09 22:39:14 Waiting for src bucket item count to get to: 200000 curr count: 157587
2021/12/09 22:39:19 Waiting for src bucket item count to get to: 200000 curr count: 174205
2021/12/09 22:39:24 Waiting for src bucket item count to get to: 200000 curr count: 189864
2021/12/09 22:39:29 Waiting for src bucket item count to get to: 200000 curr count: 189864
2021/12/09 22:39:34 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 22:39:34 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 22:39:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:39:39 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 22:39:44 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 22:39:49 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 22:39:54 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/09 22:39:54 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/09 22:39:54 No eventing-con process running
2021/12/09 22:39:54 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/09 22:39:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:40:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:40:05 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:40:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:40:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:40:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:40:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.00s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/09 22:40:27 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 22:40:27 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 22:40:27 No eventing-con process running
2021/12/09 22:40:27 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/09 22:40:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:40:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:40:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:40:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:40:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:40:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:40:46 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:40:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/09 22:40:52 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 22:40:57 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 22:41:02 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 22:41:07 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/09 22:41:07 App: TestDuplicateWithKVSwapReb got deployed
2021/12/09 22:41:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 22:41:07 Starting bucket ops to source bucket
2021/12/09 22:41:07 Removing node(s): [] from the cluster
2021/12/09 22:41:12 Rebalance progress: 16.66666666666667
2021/12/09 22:41:17 Rebalance progress: 16.66666666666667
2021/12/09 22:41:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:41:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:41:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/09 22:41:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:41:28 Rebalance progress: 16.66666666666667
2021/12/09 22:41:33 Rebalance progress: 16.66666666666667
2021/12/09 22:41:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:41:38 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:41:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 22:41:43 Rebalance progress: 16.66666666666667
2021/12/09 22:41:48 Rebalance progress: 16.66666666666667
2021/12/09 22:41:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:41:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:41:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:41:53 Waiting for src bucket item count to get to: 200000 curr count: 67041
2021/12/09 22:41:58 Waiting for src bucket item count to get to: 200000 curr count: 67041
2021/12/09 22:42:03 Waiting for src bucket item count to get to: 200000 curr count: 82559
2021/12/09 22:42:08 Waiting for src bucket item count to get to: 200000 curr count: 82559
2021/12/09 22:42:13 Waiting for src bucket item count to get to: 200000 curr count: 98724
2021/12/09 22:42:18 Waiting for src bucket item count to get to: 200000 curr count: 98724
2021/12/09 22:42:23 Waiting for src bucket item count to get to: 200000 curr count: 115091
2021/12/09 22:42:28 Waiting for src bucket item count to get to: 200000 curr count: 115091
2021/12/09 22:42:33 Waiting for src bucket item count to get to: 200000 curr count: 131448
2021/12/09 22:42:38 Waiting for src bucket item count to get to: 200000 curr count: 131448
2021/12/09 22:42:43 Waiting for src bucket item count to get to: 200000 curr count: 148075
2021/12/09 22:42:48 Waiting for src bucket item count to get to: 200000 curr count: 148075
2021/12/09 22:42:53 Waiting for src bucket item count to get to: 200000 curr count: 164842
2021/12/09 22:42:58 Waiting for src bucket item count to get to: 200000 curr count: 164842
2021/12/09 22:43:03 Waiting for src bucket item count to get to: 200000 curr count: 164842
2021/12/09 22:43:08 Waiting for src bucket item count to get to: 200000 curr count: 180429
2021/12/09 22:43:13 Waiting for src bucket item count to get to: 200000 curr count: 180429
2021/12/09 22:43:18 Waiting for src bucket item count to get to: 200000 curr count: 196677
2021/12/09 22:43:23 Waiting for src bucket item count to get to: 200000 curr count: 196677
2021/12/09 22:43:28 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 22:43:28 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 22:43:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:43:33 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 22:43:38 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 22:43:43 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 22:43:48 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/09 22:43:48 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/09 22:43:48 No eventing-con process running
2021/12/09 22:43:48 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/09 22:43:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:44:00 Waiting for dst bucket item count to get to: 0 curr count: 14839
2021/12/09 22:44:05 Waiting for dst bucket item count to get to: 0 curr count: 14839
2021/12/09 22:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:44:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:44:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:44:17 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/09 22:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.55s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/09 22:44:27 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 22:44:27 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 22:44:27 No eventing-con process running
2021/12/09 22:44:27 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 22:44:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:44:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:44:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:44:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:44:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:44:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:44:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:44:45 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:44:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 22:44:51 Starting bucket ops to source bucket
2021/12/09 22:44:56 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 22:45:01 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 22:45:06 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 22:45:06 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:45:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:45:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:45:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:45:12 Rebalance progress: 16.66666666666667
2021/12/09 22:45:17 Rebalance progress: 16.66666666666667
2021/12/09 22:45:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:45:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:45:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:45:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:45:27 Rebalance progress: 16.66666666666667
2021/12/09 22:45:32 Rebalance progress: 16.66666666666667
2021/12/09 22:45:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:45:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:45:37 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/09 22:45:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 22:45:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:45:47 Rebalance progress: 12.5
2021/12/09 22:45:52 Rebalance progress: 12.5
2021/12/09 22:45:57 Rebalance progress: 12.5
2021/12/09 22:46:02 Rebalance progress: 98.52941176470587
2021/12/09 22:46:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:46:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:46:12 Rebalance progress: 12.5
2021/12/09 22:46:17 Rebalance progress: 12.5
2021/12/09 22:46:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:46:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 22:46:27 Rebalance progress: 12.5
2021/12/09 22:46:32 Rebalance progress: 12.5
2021/12/09 22:46:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:46:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 22:46:42 Rebalance progress: 12.5
2021/12/09 22:46:47 Rebalance progress: 12.5
2021/12/09 22:46:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:46:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:46:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:46:58 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 22:47:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 22:47:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 22:47:13 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 22:47:13 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 22:47:13 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:47:13 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 22:47:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:47:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:47:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12100
2021/12/09 22:47:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:47:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:47:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:47:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12100
2021/12/09 22:47:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.44s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/09 22:47:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 22:47:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 22:47:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:47:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/09 22:47:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:47:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:47:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:47:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:48:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:48:05 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:48:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/09 22:48:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 22:48:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 22:48:21 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 22:48:26 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/09 22:48:26 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:48:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:48:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:48:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 22:48:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:48:32 Rebalance progress: 12.5
2021/12/09 22:48:37 Rebalance progress: 12.5
2021/12/09 22:48:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:48:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 22:48:47 Rebalance progress: 12.5
2021/12/09 22:48:52 Rebalance progress: 12.5
2021/12/09 22:48:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:48:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:48:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:49:02 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 22:49:07 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 22:49:12 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 22:49:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/09 22:49:17 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/09 22:49:17 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:49:17 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/09 22:49:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:49:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:49:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/09 22:49:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:49:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:49:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:49:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2956
2021/12/09 22:49:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2956
2021/12/09 22:49:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.38s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/09 22:49:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 22:49:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 22:49:56 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:49:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/09 22:49:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:50:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:50:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:50:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:50:14 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:50:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/09 22:50:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 22:50:25 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 22:50:30 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 22:50:35 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/09 22:50:35 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:50:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:50:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:50:41 Rebalance progress: 12.5
2021/12/09 22:50:46 Rebalance progress: 12.5
2021/12/09 22:50:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:50:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:50:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:50:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:50:56 Rebalance progress: 12.5
2021/12/09 22:51:01 Rebalance progress: 12.5
2021/12/09 22:51:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:51:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 22:51:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:51:11 Rebalance progress: 12.5
2021/12/09 22:51:16 Rebalance progress: 12.5
2021/12/09 22:51:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:51:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:51:26 Rebalance progress: 12.5
2021/12/09 22:51:31 Rebalance progress: 12.5
2021/12/09 22:51:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:51:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 22:51:41 Rebalance progress: 12.5
2021/12/09 22:51:46 Rebalance progress: 12.5
2021/12/09 22:51:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:51:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:51:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 22:51:56 Rebalance progress: 12.5
2021/12/09 22:52:01 Rebalance progress: 12.5
2021/12/09 22:52:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:52:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:52:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:52:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:52:12 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 22:52:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 22:52:22 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 22:52:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/09 22:52:27 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/09 22:52:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:52:27 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/09 22:52:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:52:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:52:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10620
2021/12/09 22:52:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:52:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:52:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:52:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3147
2021/12/09 22:52:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3147
2021/12/09 22:52:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (189.64s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/09 22:53:05 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 22:53:05 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 22:53:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:53:05 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/09 22:53:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:53:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:53:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:53:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:53:19 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:53:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 22:53:25 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 22:53:30 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 22:53:35 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 22:53:40 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/09 22:53:40 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:53:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:53:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:53:45 Rebalance progress: 12.5
2021/12/09 22:53:50 Rebalance progress: 12.5
2021/12/09 22:53:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:53:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:53:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:53:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 22:54:00 Rebalance progress: 12.5
2021/12/09 22:54:05 Rebalance progress: 12.5
2021/12/09 22:54:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:54:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:54:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 22:54:16 Rebalance progress: 12.5
2021/12/09 22:54:21 Rebalance progress: 12.5
2021/12/09 22:54:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:54:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:54:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:54:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:54:31 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 22:54:36 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 22:54:41 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 22:54:46 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/09 22:54:46 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/09 22:54:46 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:54:46 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/09 22:54:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:54:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:54:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6062
2021/12/09 22:54:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:54:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:55:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6062
2021/12/09 22:55:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (134.09s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/09 22:55:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:55:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:55:20 Rebalance progress: 100
2021/12/09 22:55:25 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 22:55:25 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 22:55:25 No eventing-con process running
2021/12/09 22:55:25 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/09 22:55:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:55:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:55:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:55:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:55:38 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:55:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 22:55:44 Starting bucket ops to source bucket
2021/12/09 22:55:49 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 22:55:54 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 22:55:59 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/09 22:55:59 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:55:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:55:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:56:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:56:05 Rebalance progress: 12.5
2021/12/09 22:56:10 Rebalance progress: 12.5
2021/12/09 22:56:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:56:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 22:56:15 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 22:56:20 Rebalance progress: 12.5
2021/12/09 22:56:25 Rebalance progress: 12.5
2021/12/09 22:56:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:56:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:56:36 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 22:56:41 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 22:56:46 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 22:56:51 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/09 22:56:51 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/09 22:56:51 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:56:51 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/09 22:56:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:56:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:57:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/09 22:57:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/09 22:57:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:57:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:57:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:57:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/09 22:57:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.81s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/09 22:57:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 22:57:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:57:29 Rebalance progress: 100
2021/12/09 22:57:34 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 22:57:34 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 22:57:34 No eventing-con process running
2021/12/09 22:57:34 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/09 22:57:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:57:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:57:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:57:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:57:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:57:48 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:57:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/09 22:57:54 Starting bucket ops to source bucket
2021/12/09 22:57:59 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 22:58:04 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 22:58:09 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/09 22:58:09 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:58:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 22:58:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 22:58:14 Rebalance progress: 12.5
2021/12/09 22:58:19 Rebalance progress: 12.5
2021/12/09 22:58:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:58:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/09 22:58:25 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 22:58:30 Rebalance progress: 12.5
2021/12/09 22:58:35 Rebalance progress: 12.5
2021/12/09 22:58:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:58:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 22:58:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 22:58:40 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/09 22:58:45 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 22:58:50 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 22:58:55 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 22:59:00 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/09 22:59:00 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/09 22:59:00 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:59:00 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/09 22:59:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37956
2021/12/09 22:59:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37956
2021/12/09 22:59:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:59:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37956
2021/12/09 22:59:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.20s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/09 22:59:38 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 22:59:38 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 22:59:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 22:59:38 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/09 22:59:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 22:59:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:57 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:59:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 22:59:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 22:59:58 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 22:59:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 23:00:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:00:09 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 23:00:14 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 23:00:19 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/09 23:00:19 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/09 23:00:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 23:00:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 23:00:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/09 23:00:34 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:00:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:00:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:00:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:00:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:00:40 Rebalance progress: 12.5
2021/12/09 23:00:45 Rebalance progress: 12.5
2021/12/09 23:00:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:00:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:00:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:00:50 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:00:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:00:55 Rebalance progress: 12.5
2021/12/09 23:01:00 Rebalance progress: 12.5
2021/12/09 23:01:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:01:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 23:01:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:01:11 Rebalance progress: 12.5
2021/12/09 23:01:16 Rebalance progress: 12.5
2021/12/09 23:01:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:01:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:01:26 Rebalance progress: 12.5
2021/12/09 23:01:31 Rebalance progress: 12.5
2021/12/09 23:01:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:01:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:01:41 Rebalance progress: 12.5
2021/12/09 23:01:46 Rebalance progress: 12.5
2021/12/09 23:01:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:01:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:01:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/09 23:01:56 Rebalance progress: 12.5
2021/12/09 23:02:01 Rebalance progress: 12.5
2021/12/09 23:02:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:02:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:02:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:02:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 23:02:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 23:02:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 23:02:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/09 23:02:27 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/09 23:02:27 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:02:27 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/09 23:02:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:02:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:02:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:02:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:02:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:02:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12213
2021/12/09 23:02:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:02:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:02:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 23:02:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 23:03:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 23:03:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/09 23:03:07 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/09 23:03:07 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:03:07 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/09 23:03:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:03:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.05s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/09 23:03:30 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 23:03:30 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 23:03:30 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:03:30 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/09 23:03:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:03:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:49 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:03:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:03:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:03:50 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:03:51 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/09 23:03:56 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:04:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 23:04:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 23:04:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/09 23:04:11 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/09 23:04:16 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 23:04:21 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 23:04:26 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/09 23:04:26 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:04:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:04:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:04:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/09 23:04:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:04:32 Rebalance progress: 12.5
2021/12/09 23:04:37 Rebalance progress: 12.5
2021/12/09 23:04:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:04:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/09 23:04:47 Rebalance progress: 12.5
2021/12/09 23:04:52 Rebalance progress: 12.5
2021/12/09 23:04:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:04:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:04:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:05:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 23:05:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 23:05:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 23:05:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/09 23:05:18 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/09 23:05:18 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:05:18 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/09 23:05:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:05:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:05:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2047
2021/12/09 23:05:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2047
2021/12/09 23:05:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:05:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:05:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:05:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6114
2021/12/09 23:05:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:05:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:05:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 23:05:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 23:06:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 23:06:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/09 23:06:08 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/09 23:06:08 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:06:08 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/09 23:06:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:06:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:06:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:06:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:06:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:06:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.34s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/09 23:06:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 23:06:32 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 23:06:32 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:06:32 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/09 23:06:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:06:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:06:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:06:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:06:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:06:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:06:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 23:06:50 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 23:06:50 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:06:50 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/09 23:06:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:06:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:06:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:06:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:07:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:07:04 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:07:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:07:05 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:07:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/09 23:07:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:07:16 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 23:07:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 23:07:26 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/09 23:07:26 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/09 23:07:31 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 23:07:36 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 23:07:41 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/09 23:07:41 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:07:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:07:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:07:46 Rebalance progress: 12.5
2021/12/09 23:07:51 Rebalance progress: 12.5
2021/12/09 23:07:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:07:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:07:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:07:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:08:02 Rebalance progress: 12.5
2021/12/09 23:08:07 Rebalance progress: 12.5
2021/12/09 23:08:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:08:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:08:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:08:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:08:17 Rebalance progress: 12.5
2021/12/09 23:08:22 Rebalance progress: 12.5
2021/12/09 23:08:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:08:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:08:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:08:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:08:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 23:08:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 23:08:42 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 23:08:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/09 23:08:47 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/09 23:08:47 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:08:47 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/09 23:08:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:08:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:09:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1517
2021/12/09 23:09:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1517
2021/12/09 23:09:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:09:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:09:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:09:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7607
2021/12/09 23:09:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:09:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:09:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 23:09:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 23:09:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 23:09:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/09 23:09:37 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/09 23:09:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:09:38 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/09 23:09:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:09:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:09:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:09:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:09:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:09:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.32s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 23:10:01 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:10:01 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 23:10:01 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:10:01 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 23:10:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:10:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:10:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:10:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:10:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:10:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:10:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:10:20 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:10:21 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 23:10:26 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:10:31 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:10:36 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:10:41 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:10:41 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:10:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:10:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:10:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:10:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:11:01 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:11:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:11:37 Rebalance progress: 12.5
2021/12/09 23:11:42 Rebalance progress: 12.5
2021/12/09 23:11:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:11:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:11:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:11:53 Rebalance progress: 12.5
2021/12/09 23:11:58 Rebalance progress: 12.5
2021/12/09 23:12:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:12:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:12:08 Rebalance progress: 12.5
2021/12/09 23:12:13 Rebalance progress: 12.5
2021/12/09 23:12:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:12:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:12:23 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:12:28 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:12:33 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:12:38 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:12:38 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 23:12:38 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:12:38 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 23:12:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:12:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:12:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:12:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:12:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11179
2021/12/09 23:12:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.29s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/09 23:13:06 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:13:06 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 23:13:06 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:13:06 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/09 23:13:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:13:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:13:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:13:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:13:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:13:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:13:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:13:25 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:13:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/09 23:13:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:13:36 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:13:41 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:13:46 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/09 23:13:46 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:13:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:13:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:13:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:13:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:14:06 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:14:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:14:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:14:37 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:14:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:14:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:14:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:15:08 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:15:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:15:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:15:39 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:15:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:15:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:15:50 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:16:10 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:16:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:16:25 Rebalance progress: 12.5
2021/12/09 23:16:30 Rebalance progress: 12.5
2021/12/09 23:16:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:16:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:16:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:16:41 Rebalance progress: 12.5
2021/12/09 23:16:46 Rebalance progress: 12.5
2021/12/09 23:16:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:16:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:16:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:16:56 Rebalance progress: 12.5
2021/12/09 23:17:01 Rebalance progress: 12.5
2021/12/09 23:17:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:17:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:17:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:17:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:17:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:17:16 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:17:21 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:17:26 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/09 23:17:26 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/09 23:17:26 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:17:26 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/09 23:17:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:17:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:17:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21483
2021/12/09 23:17:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:17:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:17:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:17:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (288.27s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/09 23:17:55 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 23:17:55 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 23:17:55 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:17:55 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/09 23:17:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:18:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:18:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:18:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:18:13 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:18:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 23:18:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:18:24 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 23:18:29 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 23:18:34 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/09 23:18:34 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:18:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:18:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:18:40 Rebalance progress: 12.5
2021/12/09 23:18:45 Rebalance progress: 12.5
2021/12/09 23:18:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:18:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:18:50 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:19:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:19:05 Rebalance progress: 12.5
2021/12/09 23:19:10 Rebalance progress: 12.5
2021/12/09 23:19:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:19:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:19:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:19:20 Rebalance progress: 12.5
2021/12/09 23:19:25 Rebalance progress: 12.5
2021/12/09 23:19:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:19:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:19:35 Rebalance progress: 12.5
2021/12/09 23:19:40 Rebalance progress: 12.5
2021/12/09 23:19:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:19:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:19:50 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 23:19:55 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 23:20:00 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 23:20:05 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/09 23:20:05 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/09 23:20:05 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:20:06 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/09 23:20:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:20:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:20:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8609
2021/12/09 23:20:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:20:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:20:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:20:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8609
2021/12/09 23:20:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (164.41s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/09 23:20:39 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 23:20:39 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 23:20:39 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:20:39 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/09 23:20:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:20:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:20:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:20:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:20:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:20:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:20:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:20:57 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:20:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/09 23:21:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:21:09 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 23:21:14 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 23:21:19 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/09 23:21:19 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:21:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/09 23:21:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:21:24 Rebalance progress: 12.5
2021/12/09 23:21:29 Rebalance progress: 12.5
2021/12/09 23:21:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:21:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/09 23:21:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:21:39 Rebalance progress: 12.5
2021/12/09 23:21:44 Rebalance progress: 12.5
2021/12/09 23:21:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:21:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:21:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:21:54 Rebalance progress: 12.5
2021/12/09 23:21:59 Rebalance progress: 12.5
2021/12/09 23:22:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:22:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:22:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:22:10 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 23:22:15 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 23:22:20 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 23:22:25 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/09 23:22:25 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/09 23:22:25 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:22:25 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/09 23:22:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:22:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:22:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6077
2021/12/09 23:22:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:22:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:22:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:22:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6077
2021/12/09 23:22:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.14s)
=== RUN   TestEventingRebInWithOps
2021/12/09 23:22:58 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 23:22:58 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/09 23:22:58 eventingSetup.go:46: No eventing-con process running
2021/12/09 23:22:58 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/09 23:23:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:23:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:23:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:23:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:23:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:23:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:23:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:23:17 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:23:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 23:23:23 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 23:23:28 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 23:23:33 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 23:23:38 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/09 23:23:38 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/09 23:23:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:23:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/09 23:23:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/09 23:23:43 Rebalance progress: 12.5
2021/12/09 23:23:48 Rebalance progress: 12.5
2021/12/09 23:23:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:23:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:23:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:23:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:23:58 Rebalance progress: 12.5
2021/12/09 23:24:03 Rebalance progress: 12.5
2021/12/09 23:24:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:24:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:24:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:24:08 Waiting for src bucket item count to get to: 200000 curr count: 33258
2021/12/09 23:24:13 Waiting for src bucket item count to get to: 200000 curr count: 49587
2021/12/09 23:24:18 Waiting for src bucket item count to get to: 200000 curr count: 49587
2021/12/09 23:24:23 Waiting for src bucket item count to get to: 200000 curr count: 66090
2021/12/09 23:24:28 Waiting for src bucket item count to get to: 200000 curr count: 66090
2021/12/09 23:24:33 Waiting for src bucket item count to get to: 200000 curr count: 80965
2021/12/09 23:24:38 Waiting for src bucket item count to get to: 200000 curr count: 80965
2021/12/09 23:24:43 Waiting for src bucket item count to get to: 200000 curr count: 96893
2021/12/09 23:24:48 Waiting for src bucket item count to get to: 200000 curr count: 96893
2021/12/09 23:24:53 Waiting for src bucket item count to get to: 200000 curr count: 96893
2021/12/09 23:24:58 Waiting for src bucket item count to get to: 200000 curr count: 113147
2021/12/09 23:25:03 Waiting for src bucket item count to get to: 200000 curr count: 113147
2021/12/09 23:25:08 Waiting for src bucket item count to get to: 200000 curr count: 129371
2021/12/09 23:25:13 Waiting for src bucket item count to get to: 200000 curr count: 129371
2021/12/09 23:25:18 Waiting for src bucket item count to get to: 200000 curr count: 146091
2021/12/09 23:25:23 Waiting for src bucket item count to get to: 200000 curr count: 146091
2021/12/09 23:25:28 Waiting for src bucket item count to get to: 200000 curr count: 162366
2021/12/09 23:25:33 Waiting for src bucket item count to get to: 200000 curr count: 162366
2021/12/09 23:25:38 Waiting for src bucket item count to get to: 200000 curr count: 177749
2021/12/09 23:25:43 Waiting for src bucket item count to get to: 200000 curr count: 177749
2021/12/09 23:25:48 Waiting for src bucket item count to get to: 200000 curr count: 193982
2021/12/09 23:25:53 Waiting for src bucket item count to get to: 200000 curr count: 193982
2021/12/09 23:25:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 23:25:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 23:25:58 Starting bucket ops to source bucket
2021/12/09 23:29:43 Waiting for dst bucket item count to get to: 0 curr count: 16668
2021/12/09 23:29:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:29:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 23:29:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:29:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 23:29:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 23:29:58 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 23:30:04 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/09 23:30:04 App: TestEventingRebInWithOps got un-deployed
2021/12/09 23:30:04 No eventing-con process running
2021/12/09 23:30:04 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/09 23:30:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:30:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:30:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:30:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:30:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:30:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (448.89s)
=== RUN   TestEventingRebOutWithOps
2021/12/09 23:30:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:30:22 Removing node(s): [] from the cluster
2021/12/09 23:30:28 Rebalance progress: 100
2021/12/09 23:30:33 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 23:30:33 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 23:30:33 No eventing-con process running
2021/12/09 23:30:33 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/09 23:30:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:30:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:30:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:30:51 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:30:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 23:30:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 23:31:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 23:31:07 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 23:31:12 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/09 23:31:12 App: TestEventingRebOutWithOps got deployed
2021/12/09 23:31:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:31:12 Starting bucket ops to source bucket
2021/12/09 23:31:17 Rebalance progress: 12.5
2021/12/09 23:31:22 Rebalance progress: 12.5
2021/12/09 23:31:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:31:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:31:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:31:27 Waiting for src bucket item count to get to: 200000 curr count: 10844
2021/12/09 23:31:32 Waiting for src bucket item count to get to: 200000 curr count: 27055
2021/12/09 23:31:37 Waiting for src bucket item count to get to: 200000 curr count: 27055
2021/12/09 23:31:42 Waiting for src bucket item count to get to: 200000 curr count: 43217
2021/12/09 23:31:47 Waiting for src bucket item count to get to: 200000 curr count: 43217
2021/12/09 23:31:52 Waiting for src bucket item count to get to: 200000 curr count: 59217
2021/12/09 23:31:57 Waiting for src bucket item count to get to: 200000 curr count: 59217
2021/12/09 23:32:02 Waiting for src bucket item count to get to: 200000 curr count: 59217
2021/12/09 23:32:07 Waiting for src bucket item count to get to: 200000 curr count: 74365
2021/12/09 23:32:12 Waiting for src bucket item count to get to: 200000 curr count: 74365
2021/12/09 23:32:17 Waiting for src bucket item count to get to: 200000 curr count: 90085
2021/12/09 23:32:22 Waiting for src bucket item count to get to: 200000 curr count: 90085
2021/12/09 23:32:27 Waiting for src bucket item count to get to: 200000 curr count: 106311
2021/12/09 23:32:32 Waiting for src bucket item count to get to: 200000 curr count: 106311
2021/12/09 23:32:37 Waiting for src bucket item count to get to: 200000 curr count: 122433
2021/12/09 23:32:42 Waiting for src bucket item count to get to: 200000 curr count: 122433
2021/12/09 23:32:47 Waiting for src bucket item count to get to: 200000 curr count: 138691
2021/12/09 23:32:52 Waiting for src bucket item count to get to: 200000 curr count: 138691
2021/12/09 23:32:57 Waiting for src bucket item count to get to: 200000 curr count: 155227
2021/12/09 23:33:02 Waiting for src bucket item count to get to: 200000 curr count: 155227
2021/12/09 23:33:07 Waiting for src bucket item count to get to: 200000 curr count: 170860
2021/12/09 23:33:12 Waiting for src bucket item count to get to: 200000 curr count: 170860
2021/12/09 23:33:17 Waiting for src bucket item count to get to: 200000 curr count: 186883
2021/12/09 23:33:22 Waiting for src bucket item count to get to: 200000 curr count: 186883
2021/12/09 23:33:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 23:33:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 23:33:27 Starting bucket ops to source bucket
2021/12/09 23:37:12 Waiting for dst bucket item count to get to: 0 curr count: 14519
2021/12/09 23:37:17 Waiting for dst bucket item count to get to: 0 curr count: 14519
2021/12/09 23:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:37:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 23:37:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:37:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 23:37:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 23:37:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 23:37:37 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/09 23:37:37 App: TestEventingRebOutWithOps got un-deployed
2021/12/09 23:37:37 No eventing-con process running
2021/12/09 23:37:37 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/09 23:37:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:37:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:37:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:37:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:37:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (453.91s)
=== RUN   TestEventingSwapRebWithOps
2021/12/09 23:38:01 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 23:38:01 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 23:38:01 No eventing-con process running
2021/12/09 23:38:01 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/09 23:38:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:38:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:38:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:38:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:38:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:38:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:38:19 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:38:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 23:38:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 23:38:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 23:38:36 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 23:38:41 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/09 23:38:41 App: TestEventingSwapRebWithOps got deployed
2021/12/09 23:38:41 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/09 23:38:41 Starting bucket ops to source bucket
2021/12/09 23:38:41 Removing node(s): [] from the cluster
2021/12/09 23:38:46 Rebalance progress: 12.5
2021/12/09 23:38:51 Rebalance progress: 12.5
2021/12/09 23:38:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:38:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/09 23:38:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:39:01 Rebalance progress: 12.5
2021/12/09 23:39:06 Rebalance progress: 12.5
2021/12/09 23:39:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:39:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/09 23:39:16 Rebalance progress: 12.5
2021/12/09 23:39:21 Rebalance progress: 12.5
2021/12/09 23:39:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:39:26 Waiting for src bucket item count to get to: 200000 curr count: 59873
2021/12/09 23:39:31 Waiting for src bucket item count to get to: 200000 curr count: 75063
2021/12/09 23:39:37 Waiting for src bucket item count to get to: 200000 curr count: 75063
2021/12/09 23:39:42 Waiting for src bucket item count to get to: 200000 curr count: 91298
2021/12/09 23:39:47 Waiting for src bucket item count to get to: 200000 curr count: 91298
2021/12/09 23:39:52 Waiting for src bucket item count to get to: 200000 curr count: 107784
2021/12/09 23:39:57 Waiting for src bucket item count to get to: 200000 curr count: 107784
2021/12/09 23:40:02 Waiting for src bucket item count to get to: 200000 curr count: 123965
2021/12/09 23:40:07 Waiting for src bucket item count to get to: 200000 curr count: 123965
2021/12/09 23:40:12 Waiting for src bucket item count to get to: 200000 curr count: 140244
2021/12/09 23:40:17 Waiting for src bucket item count to get to: 200000 curr count: 140244
2021/12/09 23:40:22 Waiting for src bucket item count to get to: 200000 curr count: 140244
2021/12/09 23:40:27 Waiting for src bucket item count to get to: 200000 curr count: 156559
2021/12/09 23:40:32 Waiting for src bucket item count to get to: 200000 curr count: 156559
2021/12/09 23:40:37 Waiting for src bucket item count to get to: 200000 curr count: 171670
2021/12/09 23:40:42 Waiting for src bucket item count to get to: 200000 curr count: 171670
2021/12/09 23:40:47 Waiting for src bucket item count to get to: 200000 curr count: 187637
2021/12/09 23:40:52 Waiting for src bucket item count to get to: 200000 curr count: 187637
2021/12/09 23:40:57 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 23:40:57 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 23:40:57 Starting bucket ops to source bucket
2021/12/09 23:44:42 Waiting for dst bucket item count to get to: 0 curr count: 14175
2021/12/09 23:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:44:42 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 23:44:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:44:47 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 23:44:52 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 23:44:57 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 23:45:02 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/09 23:45:02 App: TestEventingSwapRebWithOps got un-deployed
2021/12/09 23:45:02 No eventing-con process running
2021/12/09 23:45:02 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/09 23:45:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:45:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:45:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:45:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (444.90s)
=== RUN   TestKVRebInWithOps
2021/12/09 23:45:26 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 23:45:26 App: TestKVRebInWithOps got un-deployed
2021/12/09 23:45:26 No eventing-con process running
2021/12/09 23:45:26 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/09 23:45:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:45:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:45:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:45:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:45:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:45:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:45:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:45:44 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:45:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 23:45:50 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 23:45:55 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 23:46:00 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 23:46:05 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/09 23:46:05 App: TestKVRebInWithOps got deployed
2021/12/09 23:46:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 23:46:05 Starting bucket ops to source bucket
2021/12/09 23:46:06 Removing node(s): [] from the cluster
2021/12/09 23:46:11 Rebalance progress: 12.5
2021/12/09 23:46:16 Rebalance progress: 12.5
2021/12/09 23:46:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:46:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:46:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:46:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:46:26 Rebalance progress: 12.5
2021/12/09 23:46:31 Rebalance progress: 12.5
2021/12/09 23:46:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:46:36 Waiting for src bucket item count to get to: 200000 curr count: 36855
2021/12/09 23:46:41 Waiting for src bucket item count to get to: 200000 curr count: 52718
2021/12/09 23:46:46 Waiting for src bucket item count to get to: 200000 curr count: 52718
2021/12/09 23:46:51 Waiting for src bucket item count to get to: 200000 curr count: 68934
2021/12/09 23:46:56 Waiting for src bucket item count to get to: 200000 curr count: 68934
2021/12/09 23:47:01 Waiting for src bucket item count to get to: 200000 curr count: 83707
2021/12/09 23:47:06 Waiting for src bucket item count to get to: 200000 curr count: 83707
2021/12/09 23:47:11 Waiting for src bucket item count to get to: 200000 curr count: 99603
2021/12/09 23:47:16 Waiting for src bucket item count to get to: 200000 curr count: 99603
2021/12/09 23:47:21 Waiting for src bucket item count to get to: 200000 curr count: 115787
2021/12/09 23:47:26 Waiting for src bucket item count to get to: 200000 curr count: 115787
2021/12/09 23:47:31 Waiting for src bucket item count to get to: 200000 curr count: 131768
2021/12/09 23:47:36 Waiting for src bucket item count to get to: 200000 curr count: 131768
2021/12/09 23:47:41 Waiting for src bucket item count to get to: 200000 curr count: 131768
2021/12/09 23:47:46 Waiting for src bucket item count to get to: 200000 curr count: 147943
2021/12/09 23:47:51 Waiting for src bucket item count to get to: 200000 curr count: 147943
2021/12/09 23:47:56 Waiting for src bucket item count to get to: 200000 curr count: 164088
2021/12/09 23:48:01 Waiting for src bucket item count to get to: 200000 curr count: 164088
2021/12/09 23:48:06 Waiting for src bucket item count to get to: 200000 curr count: 178787
2021/12/09 23:48:11 Waiting for src bucket item count to get to: 200000 curr count: 178787
2021/12/09 23:48:16 Waiting for src bucket item count to get to: 200000 curr count: 194986
2021/12/09 23:48:21 Waiting for src bucket item count to get to: 200000 curr count: 194986
2021/12/09 23:48:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 23:48:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 23:48:26 Starting bucket ops to source bucket
2021/12/09 23:52:12 Waiting for dst bucket item count to get to: 0 curr count: 13475
2021/12/09 23:52:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:52:12 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 23:52:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:52:17 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 23:52:22 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 23:52:27 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 23:52:32 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/09 23:52:32 App: TestKVRebInWithOps got un-deployed
2021/12/09 23:52:32 No eventing-con process running
2021/12/09 23:52:32 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/09 23:52:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:52:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:52:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:52:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:52:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (449.49s)
=== RUN   TestKVRebOutWithOps
2021/12/09 23:52:50 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/09 23:52:51 Removing node(s): [] from the cluster
2021/12/09 23:52:56 Rebalance progress: 100
2021/12/09 23:53:01 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 23:53:01 App: TestKVRebOutWithOps got un-deployed
2021/12/09 23:53:01 No eventing-con process running
2021/12/09 23:53:01 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/09 23:53:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:53:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:53:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:53:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:53:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:53:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:53:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/09 23:53:19 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/09 23:53:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/09 23:53:25 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 23:53:30 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 23:53:35 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 23:53:40 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/09 23:53:40 App: TestKVRebOutWithOps got deployed
2021/12/09 23:53:40 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/09 23:53:40 Starting bucket ops to source bucket
2021/12/09 23:53:45 Rebalance progress: 12.5
2021/12/09 23:53:50 Rebalance progress: 12.5
2021/12/09 23:53:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:53:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/09 23:53:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/09 23:53:55 Waiting for src bucket item count to get to: 200000 curr count: 13909
2021/12/09 23:54:00 Waiting for src bucket item count to get to: 200000 curr count: 30135
2021/12/09 23:54:05 Waiting for src bucket item count to get to: 200000 curr count: 30135
2021/12/09 23:54:10 Waiting for src bucket item count to get to: 200000 curr count: 46590
2021/12/09 23:54:15 Waiting for src bucket item count to get to: 200000 curr count: 46590
2021/12/09 23:54:20 Waiting for src bucket item count to get to: 200000 curr count: 63025
2021/12/09 23:54:25 Waiting for src bucket item count to get to: 200000 curr count: 63025
2021/12/09 23:54:30 Waiting for src bucket item count to get to: 200000 curr count: 63025
2021/12/09 23:54:35 Waiting for src bucket item count to get to: 200000 curr count: 78217
2021/12/09 23:54:40 Waiting for src bucket item count to get to: 200000 curr count: 78217
2021/12/09 23:54:45 Waiting for src bucket item count to get to: 200000 curr count: 94646
2021/12/09 23:54:50 Waiting for src bucket item count to get to: 200000 curr count: 94646
2021/12/09 23:54:55 Waiting for src bucket item count to get to: 200000 curr count: 110817
2021/12/09 23:55:00 Waiting for src bucket item count to get to: 200000 curr count: 110817
2021/12/09 23:55:05 Waiting for src bucket item count to get to: 200000 curr count: 127013
2021/12/09 23:55:11 Waiting for src bucket item count to get to: 200000 curr count: 127013
2021/12/09 23:55:16 Waiting for src bucket item count to get to: 200000 curr count: 143055
2021/12/09 23:55:21 Waiting for src bucket item count to get to: 200000 curr count: 143055
2021/12/09 23:55:26 Waiting for src bucket item count to get to: 200000 curr count: 159162
2021/12/09 23:55:31 Waiting for src bucket item count to get to: 200000 curr count: 159162
2021/12/09 23:55:36 Waiting for src bucket item count to get to: 200000 curr count: 174433
2021/12/09 23:55:41 Waiting for src bucket item count to get to: 200000 curr count: 174433
2021/12/09 23:55:46 Waiting for src bucket item count to get to: 200000 curr count: 190239
2021/12/09 23:55:51 Waiting for src bucket item count to get to: 200000 curr count: 190239
2021/12/09 23:55:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/09 23:55:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/09 23:55:56 Starting bucket ops to source bucket
2021/12/09 23:59:41 Waiting for dst bucket item count to get to: 0 curr count: 11656
2021/12/09 23:59:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/09 23:59:41 Post DELETE ops. Expected item count: 0 got 0
2021/12/09 23:59:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/09 23:59:46 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 23:59:51 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/09 23:59:56 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 00:00:01 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/10 00:00:01 App: TestKVRebOutWithOps got un-deployed
2021/12/10 00:00:01 No eventing-con process running
2021/12/10 00:00:01 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/10 00:00:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:00:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:00:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:00:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:00:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:00:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (449.08s)
=== RUN   TestKVSwapRebWithOps
2021/12/10 00:00:24 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 00:00:24 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 00:00:24 No eventing-con process running
2021/12/10 00:00:24 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/10 00:00:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:00:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:00:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:00:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:00:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:00:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:00:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:00:43 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:00:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/10 00:00:49 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 00:00:54 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 00:00:59 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 00:01:04 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/10 00:01:04 App: TestKVSwapRebWithOps got deployed
2021/12/10 00:01:04 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:01:04 Starting bucket ops to source bucket
2021/12/10 00:01:04 Removing node(s): [] from the cluster
2021/12/10 00:01:09 Rebalance progress: 12.5
2021/12/10 00:01:14 Rebalance progress: 12.5
2021/12/10 00:01:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:01:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 00:01:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:01:25 Rebalance progress: 12.5
2021/12/10 00:01:30 Rebalance progress: 12.5
2021/12/10 00:01:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:01:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:01:40 Rebalance progress: 12.5
2021/12/10 00:01:45 Rebalance progress: 12.5
2021/12/10 00:01:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:01:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:01:50 Waiting for src bucket item count to get to: 200000 curr count: 54055
2021/12/10 00:01:55 Waiting for src bucket item count to get to: 200000 curr count: 70326
2021/12/10 00:02:00 Waiting for src bucket item count to get to: 200000 curr count: 70326
2021/12/10 00:02:05 Waiting for src bucket item count to get to: 200000 curr count: 85327
2021/12/10 00:02:10 Waiting for src bucket item count to get to: 200000 curr count: 85327
2021/12/10 00:02:15 Waiting for src bucket item count to get to: 200000 curr count: 101761
2021/12/10 00:02:20 Waiting for src bucket item count to get to: 200000 curr count: 118240
2021/12/10 00:02:25 Waiting for src bucket item count to get to: 200000 curr count: 118240
2021/12/10 00:02:30 Waiting for src bucket item count to get to: 200000 curr count: 134822
2021/12/10 00:02:35 Waiting for src bucket item count to get to: 200000 curr count: 134822
2021/12/10 00:02:40 Waiting for src bucket item count to get to: 200000 curr count: 151407
2021/12/10 00:02:45 Waiting for src bucket item count to get to: 200000 curr count: 151407
2021/12/10 00:02:50 Waiting for src bucket item count to get to: 200000 curr count: 151407
2021/12/10 00:02:55 Waiting for src bucket item count to get to: 200000 curr count: 167763
2021/12/10 00:03:00 Waiting for src bucket item count to get to: 200000 curr count: 167763
2021/12/10 00:03:05 Waiting for src bucket item count to get to: 200000 curr count: 183021
2021/12/10 00:03:10 Waiting for src bucket item count to get to: 200000 curr count: 183021
2021/12/10 00:03:15 Waiting for src bucket item count to get to: 200000 curr count: 199600
2021/12/10 00:03:20 Waiting for src bucket item count to get to: 200000 curr count: 199600
2021/12/10 00:03:25 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/10 00:03:25 Post SET ops. Expected item count: 200000 got 200000
2021/12/10 00:03:25 Starting bucket ops to source bucket
2021/12/10 00:07:10 Waiting for dst bucket item count to get to: 0 curr count: 11471
2021/12/10 00:07:15 Waiting for dst bucket item count to get to: 0 curr count: 11471
2021/12/10 00:07:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:07:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:07:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:07:21 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 00:07:26 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 00:07:31 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 00:07:36 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/10 00:07:36 App: TestKVSwapRebWithOps got un-deployed
2021/12/10 00:07:36 No eventing-con process running
2021/12/10 00:07:36 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/10 00:07:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:07:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:07:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:07:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:07:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (454.78s)
=== RUN   TestEventingRebInWithTimers
2021/12/10 00:07:59 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 00:07:59 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 00:07:59 No eventing-con process running
2021/12/10 00:07:59 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/10 00:08:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:08:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:08:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:08:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:08:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:08:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:08:18 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:08:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:08:24 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 00:08:29 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 00:08:34 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 00:08:39 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/10 00:08:39 App: TestEventingRebInWithTimers got deployed
2021/12/10 00:08:39 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:08:39 Starting bucket ops to source bucket
2021/12/10 00:08:39 Removing node(s): [] from the cluster
2021/12/10 00:08:44 Rebalance progress: 12.5
2021/12/10 00:08:49 Rebalance progress: 12.5
2021/12/10 00:08:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:08:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:08:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:08:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:09:00 Rebalance progress: 12.5
2021/12/10 00:09:05 Rebalance progress: 12.5
2021/12/10 00:09:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:09:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:09:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:09:10 Waiting for src bucket item count to get to: 50000 curr count: 26633
2021/12/10 00:09:15 Waiting for src bucket item count to get to: 50000 curr count: 35871
2021/12/10 00:09:20 Waiting for src bucket item count to get to: 50000 curr count: 35871
2021/12/10 00:09:25 Waiting for src bucket item count to get to: 50000 curr count: 48603
2021/12/10 00:09:30 Waiting for src bucket item count to get to: 50000 curr count: 48603
2021/12/10 00:09:40 Waiting for dst bucket item count to get to: 50000 curr count: 47370
2021/12/10 00:09:45 Waiting for dst bucket item count to get to: 50000 curr count: 47370
2021/12/10 00:09:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:09:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:09:45 Starting bucket ops to source bucket
2021/12/10 00:10:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:10:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:11:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:11:09 Waiting for dst bucket item count to get to: 0 curr count: 43425
2021/12/10 00:11:14 Waiting for dst bucket item count to get to: 0 curr count: 43425
2021/12/10 00:11:19 Waiting for dst bucket item count to get to: 0 curr count: 27127
2021/12/10 00:11:24 Waiting for dst bucket item count to get to: 0 curr count: 27127
2021/12/10 00:11:29 Waiting for dst bucket item count to get to: 0 curr count: 14294
2021/12/10 00:11:34 Waiting for dst bucket item count to get to: 0 curr count: 14294
2021/12/10 00:11:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:11:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:11:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:11:39 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 00:11:44 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 00:11:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 00:11:54 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/10 00:11:55 App: TestEventingRebInWithTimers got un-deployed
2021/12/10 00:11:55 No eventing-con process running
2021/12/10 00:11:55 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/10 00:11:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:12:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:12:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:12:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:12:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:12:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (258.82s)
=== RUN   TestEventingRebOutWithTimers
2021/12/10 00:12:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:12:13 Removing node(s): [] from the cluster
2021/12/10 00:12:18 Rebalance progress: 100
2021/12/10 00:12:23 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 00:12:23 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 00:12:23 No eventing-con process running
2021/12/10 00:12:23 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/10 00:12:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:12:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:12:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:12:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:12:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:12:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:12:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:12:42 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:12:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:12:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 00:12:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 00:12:58 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 00:13:03 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/10 00:13:03 App: TestEventingRebOutWithTimers got deployed
2021/12/10 00:13:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:13:03 Starting bucket ops to source bucket
2021/12/10 00:13:08 Rebalance progress: 12.5
2021/12/10 00:13:13 Rebalance progress: 12.5
2021/12/10 00:13:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:13:18 Waiting for src bucket item count to get to: 50000 curr count: 8648
2021/12/10 00:13:23 Waiting for src bucket item count to get to: 50000 curr count: 21300
2021/12/10 00:13:28 Waiting for src bucket item count to get to: 50000 curr count: 21300
2021/12/10 00:13:33 Waiting for src bucket item count to get to: 50000 curr count: 32445
2021/12/10 00:13:38 Waiting for src bucket item count to get to: 50000 curr count: 32445
2021/12/10 00:13:43 Waiting for src bucket item count to get to: 50000 curr count: 32445
2021/12/10 00:13:48 Waiting for src bucket item count to get to: 50000 curr count: 43654
2021/12/10 00:13:53 Waiting for src bucket item count to get to: 50000 curr count: 43654
2021/12/10 00:14:03 Waiting for dst bucket item count to get to: 50000 curr count: 42091
2021/12/10 00:14:08 Waiting for dst bucket item count to get to: 50000 curr count: 42091
2021/12/10 00:14:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:14:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:14:08 Starting bucket ops to source bucket
2021/12/10 00:15:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:15:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:15:28 Waiting for dst bucket item count to get to: 0 curr count: 43907
2021/12/10 00:15:33 Waiting for dst bucket item count to get to: 0 curr count: 43907
2021/12/10 00:15:38 Waiting for dst bucket item count to get to: 0 curr count: 34942
2021/12/10 00:15:43 Waiting for dst bucket item count to get to: 0 curr count: 34942
2021/12/10 00:15:48 Waiting for dst bucket item count to get to: 0 curr count: 10849
2021/12/10 00:15:53 Waiting for dst bucket item count to get to: 0 curr count: 10849
2021/12/10 00:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:15:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:15:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:15:58 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 00:16:03 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 00:16:08 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 00:16:13 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/10 00:16:13 App: TestEventingRebOutWithTimers got un-deployed
2021/12/10 00:16:13 No eventing-con process running
2021/12/10 00:16:13 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/10 00:16:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:16:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:16:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:16:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:16:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (258.44s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/10 00:16:36 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 00:16:36 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 00:16:36 No eventing-con process running
2021/12/10 00:16:36 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/10 00:16:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:16:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:16:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:16:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:16:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:16:55 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:16:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:17:01 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 00:17:06 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 00:17:11 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 00:17:16 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/10 00:17:16 App: TestEventingSwapRebWithTimers got deployed
2021/12/10 00:17:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:17:16 Starting bucket ops to source bucket
2021/12/10 00:17:16 Removing node(s): [] from the cluster
2021/12/10 00:17:21 Rebalance progress: 12.5
2021/12/10 00:17:26 Rebalance progress: 12.5
2021/12/10 00:17:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:17:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 00:17:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:17:37 Rebalance progress: 12.5
2021/12/10 00:17:42 Rebalance progress: 12.5
2021/12/10 00:17:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:17:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:17:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:17:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:17:52 Rebalance progress: 12.5
2021/12/10 00:17:57 Rebalance progress: 12.5
2021/12/10 00:18:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:18:07 Waiting for dst bucket item count to get to: 50000 curr count: 38079
2021/12/10 00:18:12 Waiting for dst bucket item count to get to: 50000 curr count: 38079
2021/12/10 00:18:17 Waiting for dst bucket item count to get to: 50000 curr count: 47415
2021/12/10 00:18:22 Waiting for dst bucket item count to get to: 50000 curr count: 47415
2021/12/10 00:18:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:18:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:18:22 Starting bucket ops to source bucket
2021/12/10 00:19:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:19:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:19:42 Waiting for dst bucket item count to get to: 0 curr count: 49131
2021/12/10 00:19:47 Waiting for dst bucket item count to get to: 0 curr count: 49131
2021/12/10 00:19:52 Waiting for dst bucket item count to get to: 0 curr count: 41993
2021/12/10 00:19:57 Waiting for dst bucket item count to get to: 0 curr count: 41993
2021/12/10 00:20:02 Waiting for dst bucket item count to get to: 0 curr count: 26630
2021/12/10 00:20:07 Waiting for dst bucket item count to get to: 0 curr count: 26630
2021/12/10 00:20:12 Waiting for dst bucket item count to get to: 0 curr count: 2653
2021/12/10 00:20:17 Waiting for dst bucket item count to get to: 0 curr count: 2653
2021/12/10 00:20:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:20:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:20:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:20:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 00:20:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 00:20:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 00:20:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/10 00:20:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/10 00:20:37 No eventing-con process running
2021/12/10 00:20:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/10 00:20:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:20:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:20:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:20:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:20:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:20:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (264.39s)
=== RUN   TestKVRebInWithTimers
2021/12/10 00:21:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 00:21:01 App: TestKVRebInWithTimers got un-deployed
2021/12/10 00:21:01 No eventing-con process running
2021/12/10 00:21:01 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/10 00:21:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:21:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:21:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:21:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:21:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:21:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:21:19 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:21:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:21:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 00:21:31 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 00:21:36 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 00:21:41 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/10 00:21:41 App: TestKVRebInWithTimers got deployed
2021/12/10 00:21:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:21:41 Starting bucket ops to source bucket
2021/12/10 00:21:41 Removing node(s): [] from the cluster
2021/12/10 00:21:46 Rebalance progress: 12.5
2021/12/10 00:21:51 Rebalance progress: 12.5
2021/12/10 00:21:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:21:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:22:01 Rebalance progress: 12.5
2021/12/10 00:22:06 Rebalance progress: 12.5
2021/12/10 00:22:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:22:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:22:11 Waiting for src bucket item count to get to: 50000 curr count: 33891
2021/12/10 00:22:16 Waiting for src bucket item count to get to: 50000 curr count: 33891
2021/12/10 00:22:21 Waiting for src bucket item count to get to: 50000 curr count: 45484
2021/12/10 00:22:26 Waiting for src bucket item count to get to: 50000 curr count: 45484
2021/12/10 00:22:36 Waiting for dst bucket item count to get to: 50000 curr count: 43146
2021/12/10 00:22:41 Waiting for dst bucket item count to get to: 50000 curr count: 43146
2021/12/10 00:22:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:22:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:22:41 Starting bucket ops to source bucket
2021/12/10 00:23:50 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:23:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:24:00 Waiting for dst bucket item count to get to: 0 curr count: 46463
2021/12/10 00:24:05 Waiting for dst bucket item count to get to: 0 curr count: 46463
2021/12/10 00:24:10 Waiting for dst bucket item count to get to: 0 curr count: 36035
2021/12/10 00:24:15 Waiting for dst bucket item count to get to: 0 curr count: 36035
2021/12/10 00:24:20 Waiting for dst bucket item count to get to: 0 curr count: 15073
2021/12/10 00:24:25 Waiting for dst bucket item count to get to: 0 curr count: 15073
2021/12/10 00:24:30 Waiting for dst bucket item count to get to: 0 curr count: 1359
2021/12/10 00:24:35 Waiting for dst bucket item count to get to: 0 curr count: 1359
2021/12/10 00:24:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:24:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:24:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:24:40 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 00:24:45 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 00:24:51 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 00:24:56 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/10 00:24:56 App: TestKVRebInWithTimers got un-deployed
2021/12/10 00:24:56 No eventing-con process running
2021/12/10 00:24:56 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/10 00:24:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:25:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:25:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:25:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:25:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:25:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (258.20s)
=== RUN   TestKVRebOutWithTimers
2021/12/10 00:25:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:25:14 Removing node(s): [] from the cluster
2021/12/10 00:25:19 Rebalance progress: 100
2021/12/10 00:25:24 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 00:25:24 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 00:25:24 No eventing-con process running
2021/12/10 00:25:24 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/10 00:25:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:25:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:25:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:25:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:25:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:25:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:25:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:25:43 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:25:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:25:49 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 00:25:54 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 00:25:59 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 00:26:04 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/10 00:26:04 App: TestKVRebOutWithTimers got deployed
2021/12/10 00:26:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:26:04 Starting bucket ops to source bucket
2021/12/10 00:26:09 Rebalance progress: 12.5
2021/12/10 00:26:14 Rebalance progress: 12.5
2021/12/10 00:26:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:26:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:26:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:26:19 Waiting for src bucket item count to get to: 50000 curr count: 6561
2021/12/10 00:26:24 Waiting for src bucket item count to get to: 50000 curr count: 20553
2021/12/10 00:26:29 Waiting for src bucket item count to get to: 50000 curr count: 20553
2021/12/10 00:26:34 Waiting for src bucket item count to get to: 50000 curr count: 32494
2021/12/10 00:26:39 Waiting for src bucket item count to get to: 50000 curr count: 32494
2021/12/10 00:26:44 Waiting for src bucket item count to get to: 50000 curr count: 43961
2021/12/10 00:26:49 Waiting for src bucket item count to get to: 50000 curr count: 43961
2021/12/10 00:26:54 Waiting for src bucket item count to get to: 50000 curr count: 43961
2021/12/10 00:27:04 Waiting for dst bucket item count to get to: 50000 curr count: 35965
2021/12/10 00:27:09 Waiting for dst bucket item count to get to: 50000 curr count: 35965
2021/12/10 00:27:14 Waiting for dst bucket item count to get to: 50000 curr count: 46218
2021/12/10 00:27:19 Waiting for dst bucket item count to get to: 50000 curr count: 46218
2021/12/10 00:27:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:27:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:27:19 Starting bucket ops to source bucket
2021/12/10 00:28:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:28:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:28:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:28:44 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:28:49 Waiting for dst bucket item count to get to: 0 curr count: 37148
2021/12/10 00:28:54 Waiting for dst bucket item count to get to: 0 curr count: 37148
2021/12/10 00:28:59 Waiting for dst bucket item count to get to: 0 curr count: 24106
2021/12/10 00:29:04 Waiting for dst bucket item count to get to: 0 curr count: 24106
2021/12/10 00:29:09 Waiting for dst bucket item count to get to: 0 curr count: 6894
2021/12/10 00:29:14 Waiting for dst bucket item count to get to: 0 curr count: 6894
2021/12/10 00:29:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:29:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:29:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:29:19 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 00:29:24 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 00:29:29 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 00:29:34 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/10 00:29:34 App: TestKVRebOutWithTimers got un-deployed
2021/12/10 00:29:34 No eventing-con process running
2021/12/10 00:29:34 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/10 00:29:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:29:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:29:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:29:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:29:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:29:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (278.49s)
=== RUN   TestKVSwapRebWithTimers
2021/12/10 00:29:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:29:58 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 00:29:58 No eventing-con process running
2021/12/10 00:29:58 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/10 00:29:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:30:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:30:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:30:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:30:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:30:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:30:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:30:16 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:30:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/10 00:30:22 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 00:30:27 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 00:30:32 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 00:30:37 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/10 00:30:37 App: TestKVSwapRebWithTimers got deployed
2021/12/10 00:30:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:30:37 Starting bucket ops to source bucket
2021/12/10 00:30:37 Removing node(s): [] from the cluster
2021/12/10 00:30:43 Rebalance progress: 12.5
2021/12/10 00:30:48 Rebalance progress: 12.5
2021/12/10 00:30:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:30:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:30:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:30:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 00:30:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:30:58 Rebalance progress: 12.5
2021/12/10 00:31:03 Rebalance progress: 12.5
2021/12/10 00:31:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:31:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:31:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:31:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:31:14 Rebalance progress: 12.5
2021/12/10 00:31:19 Rebalance progress: 12.5
2021/12/10 00:31:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:31:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:31:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:31:24 Waiting for src bucket item count to get to: 50000 curr count: 49693
2021/12/10 00:31:29 Waiting for src bucket item count to get to: 50000 curr count: 49693
2021/12/10 00:31:39 Waiting for dst bucket item count to get to: 50000 curr count: 43335
2021/12/10 00:31:44 Waiting for dst bucket item count to get to: 50000 curr count: 43335
2021/12/10 00:31:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:31:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:31:44 Starting bucket ops to source bucket
2021/12/10 00:32:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:32:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:33:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:33:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:33:13 Waiting for dst bucket item count to get to: 0 curr count: 40978
2021/12/10 00:33:18 Waiting for dst bucket item count to get to: 0 curr count: 40978
2021/12/10 00:33:23 Waiting for dst bucket item count to get to: 0 curr count: 31567
2021/12/10 00:33:28 Waiting for dst bucket item count to get to: 0 curr count: 31567
2021/12/10 00:33:33 Waiting for dst bucket item count to get to: 0 curr count: 12624
2021/12/10 00:33:38 Waiting for dst bucket item count to get to: 0 curr count: 12624
2021/12/10 00:33:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:33:38 Post DELETE ops. Expected item count: 0 got 0
2021/12/10 00:33:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:33:43 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:33:48 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:33:54 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:33:59 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:34:04 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/10 00:34:04 App: TestKVSwapRebWithTimers got un-deployed
2021/12/10 00:34:04 No eventing-con process running
2021/12/10 00:34:04 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/10 00:34:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:34:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:34:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:34:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:34:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:34:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (269.54s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/10 00:34:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:34:27 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 00:34:27 No eventing-con process running
2021/12/10 00:34:27 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/10 00:34:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:34:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:34:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:34:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:34:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:34:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:34:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:34:46 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:34:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 00:34:52 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 00:34:57 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 00:35:02 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 00:35:07 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/10 00:35:07 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/10 00:35:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:35:07 Starting bucket ops to source bucket
2021/12/10 00:35:07 Removing node(s): [] from the cluster
2021/12/10 00:35:12 Rebalance progress: 12.5
2021/12/10 00:35:17 Rebalance progress: 12.5
2021/12/10 00:35:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:35:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:35:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:35:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:35:27 Rebalance progress: 12.5
2021/12/10 00:35:32 Rebalance progress: 12.5
2021/12/10 00:35:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:35:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:35:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:35:37 Waiting for src bucket item count to get to: 50000 curr count: 28489
2021/12/10 00:35:42 Waiting for src bucket item count to get to: 50000 curr count: 39493
2021/12/10 00:35:47 Waiting for src bucket item count to get to: 50000 curr count: 39493
2021/12/10 00:35:52 Waiting for src bucket item count to get to: 50000 curr count: 49878
2021/12/10 00:35:57 Waiting for src bucket item count to get to: 50000 curr count: 49878
2021/12/10 00:36:07 Waiting for dst bucket item count to get to: 50000 curr count: 39730
2021/12/10 00:36:12 Waiting for dst bucket item count to get to: 50000 curr count: 39730
2021/12/10 00:36:17 Waiting for dst bucket item count to get to: 50000 curr count: 48312
2021/12/10 00:36:22 Waiting for dst bucket item count to get to: 50000 curr count: 48312
2021/12/10 00:36:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:36:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:36:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:36:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/10 00:36:53 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/10 00:36:53 No eventing-con process running
2021/12/10 00:36:53 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/10 00:36:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:37:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:37:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:37:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:37:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:37:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:37:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.17s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/10 00:37:21 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:37:22 Removing node(s): [] from the cluster
2021/12/10 00:37:27 Rebalance progress: 100
2021/12/10 00:37:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:37:32 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 00:37:32 No eventing-con process running
2021/12/10 00:37:32 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/10 00:37:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:37:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:37:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:37:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:37:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:37:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:37:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:37:50 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:37:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 00:37:56 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 00:38:01 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 00:38:06 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 00:38:11 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/10 00:38:11 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/10 00:38:11 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:38:11 Starting bucket ops to source bucket
2021/12/10 00:38:16 Rebalance progress: 12.5
2021/12/10 00:38:21 Rebalance progress: 12.5
2021/12/10 00:38:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:38:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:38:27 Waiting for src bucket item count to get to: 50000 curr count: 10398
2021/12/10 00:38:32 Waiting for src bucket item count to get to: 50000 curr count: 24635
2021/12/10 00:38:37 Waiting for src bucket item count to get to: 50000 curr count: 24635
2021/12/10 00:38:42 Waiting for src bucket item count to get to: 50000 curr count: 37728
2021/12/10 00:38:47 Waiting for src bucket item count to get to: 50000 curr count: 37728
2021/12/10 00:38:52 Waiting for src bucket item count to get to: 50000 curr count: 37728
2021/12/10 00:38:57 Waiting for src bucket item count to get to: 50000 curr count: 46952
2021/12/10 00:39:02 Waiting for src bucket item count to get to: 50000 curr count: 46952
2021/12/10 00:39:12 Waiting for dst bucket item count to get to: 50000 curr count: 31304
2021/12/10 00:39:17 Waiting for dst bucket item count to get to: 50000 curr count: 31304
2021/12/10 00:39:22 Waiting for dst bucket item count to get to: 50000 curr count: 44272
2021/12/10 00:39:27 Waiting for dst bucket item count to get to: 50000 curr count: 44272
2021/12/10 00:39:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:39:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:39:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:39:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/10 00:39:57 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/10 00:39:57 No eventing-con process running
2021/12/10 00:39:57 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/10 00:39:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:40:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:40:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:40:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:40:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:40:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:40:20 Waiting for dst bucket item count to get to: 0 curr count: 26399
2021/12/10 00:40:25 Waiting for dst bucket item count to get to: 0 curr count: 26399
2021/12/10 00:40:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.22s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/10 00:40:35 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:40:35 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 00:40:35 No eventing-con process running
2021/12/10 00:40:35 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/10 00:40:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:40:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:40:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:40:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:40:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:40:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:40:54 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:40:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 00:41:00 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 00:41:05 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 00:41:10 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 00:41:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/10 00:41:15 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/10 00:41:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:41:15 Starting bucket ops to source bucket
2021/12/10 00:41:16 Removing node(s): [] from the cluster
2021/12/10 00:41:21 Rebalance progress: 12.5
2021/12/10 00:41:26 Rebalance progress: 12.5
2021/12/10 00:41:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:41:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:41:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:41:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 00:41:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:41:36 Rebalance progress: 12.5
2021/12/10 00:41:41 Rebalance progress: 12.5
2021/12/10 00:41:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:41:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:41:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:41:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:41:51 Rebalance progress: 12.5
2021/12/10 00:41:56 Rebalance progress: 12.5
2021/12/10 00:42:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:42:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:42:06 Waiting for dst bucket item count to get to: 50000 curr count: 27585
2021/12/10 00:42:11 Waiting for dst bucket item count to get to: 50000 curr count: 27585
2021/12/10 00:42:16 Waiting for dst bucket item count to get to: 50000 curr count: 36067
2021/12/10 00:42:21 Waiting for dst bucket item count to get to: 50000 curr count: 36067
2021/12/10 00:42:26 Waiting for dst bucket item count to get to: 50000 curr count: 49031
2021/12/10 00:42:31 Waiting for dst bucket item count to get to: 50000 curr count: 49031
2021/12/10 00:42:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:42:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:42:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:42:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:42:41 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:42:46 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:42:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:42:57 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:43:02 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/10 00:43:02 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/10 00:43:02 No eventing-con process running
2021/12/10 00:43:02 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/10 00:43:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:43:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:43:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:43:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:43:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:43:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:43:30 Waiting for dst bucket item count to get to: 0 curr count: 10152
2021/12/10 00:43:35 Waiting for dst bucket item count to get to: 0 curr count: 10152
2021/12/10 00:43:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (189.67s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/10 00:43:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:43:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 00:43:45 No eventing-con process running
2021/12/10 00:43:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/10 00:43:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:43:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:43:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:43:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:43:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:43:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:44:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:44:04 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:44:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 00:44:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 00:44:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 00:44:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 00:44:25 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/10 00:44:25 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/10 00:44:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:44:25 Starting bucket ops to source bucket
2021/12/10 00:44:26 Removing node(s): [] from the cluster
2021/12/10 00:44:31 Rebalance progress: 12.5
2021/12/10 00:44:36 Rebalance progress: 12.5
2021/12/10 00:44:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:44:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:44:46 Rebalance progress: 12.5
2021/12/10 00:44:51 Rebalance progress: 12.5
2021/12/10 00:44:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:44:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:44:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:44:56 Waiting for src bucket item count to get to: 50000 curr count: 32256
2021/12/10 00:45:01 Waiting for src bucket item count to get to: 50000 curr count: 43919
2021/12/10 00:45:06 Waiting for src bucket item count to get to: 50000 curr count: 43919
2021/12/10 00:45:16 Waiting for dst bucket item count to get to: 50000 curr count: 26765
2021/12/10 00:45:21 Waiting for dst bucket item count to get to: 50000 curr count: 26765
2021/12/10 00:45:26 Waiting for dst bucket item count to get to: 50000 curr count: 36282
2021/12/10 00:45:31 Waiting for dst bucket item count to get to: 50000 curr count: 36282
2021/12/10 00:45:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:45:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:45:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:45:36 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:45:41 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:45:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:45:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:45:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:46:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/10 00:46:01 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/10 00:46:01 No eventing-con process running
2021/12/10 00:46:01 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/10 00:46:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:46:13 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:46:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:46:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:46:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:46:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (169.66s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/10 00:46:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:46:30 Removing node(s): [] from the cluster
2021/12/10 00:46:35 Rebalance progress: 100
2021/12/10 00:46:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:46:40 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 00:46:40 No eventing-con process running
2021/12/10 00:46:40 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/10 00:46:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:46:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:46:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:46:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:46:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:46:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:46:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:46:58 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:47:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 00:47:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 00:47:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 00:47:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 00:47:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/10 00:47:20 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/10 00:47:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:47:20 Starting bucket ops to source bucket
2021/12/10 00:47:25 Rebalance progress: 12.5
2021/12/10 00:47:30 Rebalance progress: 12.5
2021/12/10 00:47:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:47:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:47:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:47:35 Waiting for src bucket item count to get to: 50000 curr count: 12274
2021/12/10 00:47:40 Waiting for src bucket item count to get to: 50000 curr count: 12274
2021/12/10 00:47:45 Waiting for src bucket item count to get to: 50000 curr count: 26397
2021/12/10 00:47:50 Waiting for src bucket item count to get to: 50000 curr count: 38889
2021/12/10 00:47:55 Waiting for src bucket item count to get to: 50000 curr count: 38889
2021/12/10 00:48:00 Waiting for src bucket item count to get to: 50000 curr count: 49544
2021/12/10 00:48:05 Waiting for src bucket item count to get to: 50000 curr count: 49544
2021/12/10 00:48:10 Waiting for src bucket item count to get to: 50000 curr count: 49544
2021/12/10 00:48:20 Waiting for dst bucket item count to get to: 50000 curr count: 33661
2021/12/10 00:48:25 Waiting for dst bucket item count to get to: 50000 curr count: 33661
2021/12/10 00:48:30 Waiting for dst bucket item count to get to: 50000 curr count: 44699
2021/12/10 00:48:35 Waiting for dst bucket item count to get to: 50000 curr count: 44699
2021/12/10 00:48:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:48:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:48:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:48:40 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:48:45 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:48:50 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:48:55 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:49:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:49:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/10 00:49:05 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/10 00:49:05 No eventing-con process running
2021/12/10 00:49:05 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/10 00:49:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:49:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:49:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:49:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:49:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:49:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:49:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (184.07s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/10 00:49:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:49:39 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 00:49:39 No eventing-con process running
2021/12/10 00:49:39 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/10 00:49:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:49:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:49:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:49:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:49:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:49:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:49:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:49:57 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:49:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 00:50:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 00:50:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 00:50:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 00:50:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/10 00:50:19 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/10 00:50:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 00:50:19 Starting bucket ops to source bucket
2021/12/10 00:50:19 Removing node(s): [] from the cluster
2021/12/10 00:50:24 Rebalance progress: 12.5
2021/12/10 00:50:29 Rebalance progress: 12.5
2021/12/10 00:50:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:50:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:50:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:50:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 00:50:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:50:40 Rebalance progress: 12.5
2021/12/10 00:50:45 Rebalance progress: 12.5
2021/12/10 00:50:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:50:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:50:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:50:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:50:55 Rebalance progress: 12.5
2021/12/10 00:51:00 Rebalance progress: 12.5
2021/12/10 00:51:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:51:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:51:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:51:05 Waiting for src bucket item count to get to: 50000 curr count: 48262
2021/12/10 00:51:10 Waiting for src bucket item count to get to: 50000 curr count: 48262
2021/12/10 00:51:20 Waiting for dst bucket item count to get to: 50000 curr count: 36698
2021/12/10 00:51:25 Waiting for dst bucket item count to get to: 50000 curr count: 36698
2021/12/10 00:51:30 Waiting for dst bucket item count to get to: 50000 curr count: 44059
2021/12/10 00:51:35 Waiting for dst bucket item count to get to: 50000 curr count: 44059
2021/12/10 00:51:35 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:51:35 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:51:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:51:40 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:51:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:51:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:51:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:52:00 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:52:05 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/10 00:52:05 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/10 00:52:05 No eventing-con process running
2021/12/10 00:52:05 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/10 00:52:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:52:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:52:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:52:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:52:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:52:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.82s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/10 00:52:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:52:39 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 00:52:39 No eventing-con process running
2021/12/10 00:52:39 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/10 00:52:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:52:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:52:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:52:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:52:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:52:57 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:52:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 00:53:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 00:53:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 00:53:13 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 00:53:18 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/10 00:53:18 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/10 00:53:18 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:53:18 Starting bucket ops to source bucket
2021/12/10 00:53:19 Removing node(s): [] from the cluster
2021/12/10 00:53:24 Rebalance progress: 12.5
2021/12/10 00:53:29 Rebalance progress: 12.5
2021/12/10 00:53:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:53:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:53:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:53:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:53:39 Rebalance progress: 12.5
2021/12/10 00:53:44 Rebalance progress: 12.5
2021/12/10 00:53:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:53:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:53:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:53:49 Waiting for src bucket item count to get to: 50000 curr count: 25128
2021/12/10 00:53:54 Waiting for src bucket item count to get to: 50000 curr count: 34925
2021/12/10 00:53:59 Waiting for src bucket item count to get to: 50000 curr count: 34925
2021/12/10 00:54:04 Waiting for src bucket item count to get to: 50000 curr count: 47297
2021/12/10 00:54:09 Waiting for src bucket item count to get to: 50000 curr count: 47297
2021/12/10 00:54:19 Waiting for dst bucket item count to get to: 50000 curr count: 47539
2021/12/10 00:54:24 Waiting for dst bucket item count to get to: 50000 curr count: 47539
2021/12/10 00:54:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:54:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:54:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:54:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:40 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:50 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:55 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/10 00:54:55 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/10 00:54:55 No eventing-con process running
2021/12/10 00:54:55 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/10 00:54:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:55:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:55:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:55:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:55:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:55:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:55:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.44s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/10 00:55:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:55:23 Removing node(s): [] from the cluster
2021/12/10 00:55:28 Rebalance progress: 100
2021/12/10 00:55:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:55:33 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 00:55:33 No eventing-con process running
2021/12/10 00:55:33 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/10 00:55:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:55:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:55:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:55:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:55:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:55:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:55:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:55:52 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:55:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 00:55:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 00:56:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 00:56:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 00:56:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/10 00:56:13 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/10 00:56:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:56:13 Starting bucket ops to source bucket
2021/12/10 00:56:18 Rebalance progress: 12.5
2021/12/10 00:56:23 Rebalance progress: 12.5
2021/12/10 00:56:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:56:28 Waiting for src bucket item count to get to: 50000 curr count: 8161
2021/12/10 00:56:33 Waiting for src bucket item count to get to: 50000 curr count: 20582
2021/12/10 00:56:38 Waiting for src bucket item count to get to: 50000 curr count: 20582
2021/12/10 00:56:43 Waiting for src bucket item count to get to: 50000 curr count: 31611
2021/12/10 00:56:48 Waiting for src bucket item count to get to: 50000 curr count: 31611
2021/12/10 00:56:53 Waiting for src bucket item count to get to: 50000 curr count: 31611
2021/12/10 00:56:58 Waiting for src bucket item count to get to: 50000 curr count: 43098
2021/12/10 00:57:03 Waiting for src bucket item count to get to: 50000 curr count: 43098
2021/12/10 00:57:13 Waiting for dst bucket item count to get to: 50000 curr count: 41782
2021/12/10 00:57:18 Waiting for dst bucket item count to get to: 50000 curr count: 41782
2021/12/10 00:57:18 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 00:57:18 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 00:57:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 00:57:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:29 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:34 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/10 00:57:49 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/10 00:57:49 No eventing-con process running
2021/12/10 00:57:49 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/10 00:57:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:57:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:58:01 Waiting for dst bucket item count to get to: 0 curr count: 31675
2021/12/10 00:58:06 Waiting for dst bucket item count to get to: 0 curr count: 31675
2021/12/10 00:58:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:58:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:58:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 00:58:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.28s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/10 00:58:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 00:58:27 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 00:58:27 No eventing-con process running
2021/12/10 00:58:27 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/10 00:58:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:58:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:58:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 00:58:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:58:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:58:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:58:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 00:58:46 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 00:58:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 00:58:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 00:58:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 00:59:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 00:59:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/10 00:59:07 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/10 00:59:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/10 00:59:07 Starting bucket ops to source bucket
2021/12/10 00:59:08 Removing node(s): [] from the cluster
2021/12/10 00:59:13 Rebalance progress: 12.5
2021/12/10 00:59:18 Rebalance progress: 12.5
2021/12/10 00:59:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:59:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 00:59:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 00:59:28 Rebalance progress: 12.5
2021/12/10 00:59:33 Rebalance progress: 12.5
2021/12/10 00:59:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:59:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 00:59:44 Rebalance progress: 12.5
2021/12/10 00:59:49 Rebalance progress: 12.5
2021/12/10 00:59:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 00:59:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 00:59:54 Waiting for src bucket item count to get to: 50000 curr count: 48666
2021/12/10 00:59:59 Waiting for src bucket item count to get to: 50000 curr count: 48666
2021/12/10 01:00:09 Waiting for dst bucket item count to get to: 50000 curr count: 42011
2021/12/10 01:00:14 Waiting for dst bucket item count to get to: 50000 curr count: 42011
2021/12/10 01:00:19 Waiting for dst bucket item count to get to: 50000 curr count: 48576
2021/12/10 01:00:24 Waiting for dst bucket item count to get to: 50000 curr count: 48576
2021/12/10 01:00:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 01:00:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 01:00:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:00:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/10 01:00:54 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/10 01:00:54 No eventing-con process running
2021/12/10 01:00:54 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/10 01:00:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:01:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:01:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 01:01:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:01:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:01:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:01:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 01:01:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (180.00s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/10 01:01:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:01:27 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 01:01:27 No eventing-con process running
2021/12/10 01:01:27 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/10 01:01:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:01:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:01:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:01:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:01:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:01:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:01:46 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:01:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/10 01:01:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 01:01:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 01:02:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 01:02:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/10 01:02:07 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/10 01:02:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:02:07 Starting bucket ops to source bucket
2021/12/10 01:02:07 Removing node(s): [] from the cluster
2021/12/10 01:02:13 Rebalance progress: 12.5
2021/12/10 01:02:18 Rebalance progress: 12.5
2021/12/10 01:02:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:02:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:02:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:02:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:02:28 Rebalance progress: 12.5
2021/12/10 01:02:33 Rebalance progress: 12.5
2021/12/10 01:02:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:02:38 Waiting for src bucket item count to get to: 50000 curr count: 28653
2021/12/10 01:02:43 Waiting for src bucket item count to get to: 50000 curr count: 38395
2021/12/10 01:02:48 Waiting for src bucket item count to get to: 50000 curr count: 38395
2021/12/10 01:02:58 Waiting for dst bucket item count to get to: 50000 curr count: 30800
2021/12/10 01:03:03 Waiting for dst bucket item count to get to: 50000 curr count: 30800
2021/12/10 01:03:08 Waiting for dst bucket item count to get to: 50000 curr count: 44201
2021/12/10 01:03:13 Waiting for dst bucket item count to get to: 50000 curr count: 44201
2021/12/10 01:03:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 01:03:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 01:03:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:03:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:03:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:03:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:03:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:03:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/10 01:03:38 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/10 01:03:38 No eventing-con process running
2021/12/10 01:03:38 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/10 01:03:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:03:50 Waiting for dst bucket item count to get to: 0 curr count: 2928
2021/12/10 01:03:55 Waiting for dst bucket item count to get to: 0 curr count: 2928
2021/12/10 01:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:03:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:04:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:04:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 01:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.37s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/10 01:04:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:04:12 Removing node(s): [] from the cluster
2021/12/10 01:04:17 Rebalance progress: 100
2021/12/10 01:04:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:04:22 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 01:04:22 No eventing-con process running
2021/12/10 01:04:22 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/10 01:04:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:04:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:04:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:04:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:04:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:04:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:04:41 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:04:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/10 01:04:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 01:04:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 01:04:57 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 01:05:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/10 01:05:02 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/10 01:05:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:05:02 Starting bucket ops to source bucket
2021/12/10 01:05:07 Rebalance progress: 12.5
2021/12/10 01:05:12 Rebalance progress: 12.5
2021/12/10 01:05:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:05:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:05:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:05:17 Waiting for src bucket item count to get to: 50000 curr count: 10211
2021/12/10 01:05:22 Waiting for src bucket item count to get to: 50000 curr count: 23616
2021/12/10 01:05:27 Waiting for src bucket item count to get to: 50000 curr count: 23616
2021/12/10 01:05:32 Waiting for src bucket item count to get to: 50000 curr count: 34828
2021/12/10 01:05:37 Waiting for src bucket item count to get to: 50000 curr count: 34828
2021/12/10 01:05:42 Waiting for src bucket item count to get to: 50000 curr count: 34828
2021/12/10 01:05:47 Waiting for src bucket item count to get to: 50000 curr count: 45173
2021/12/10 01:05:52 Waiting for src bucket item count to get to: 50000 curr count: 45173
2021/12/10 01:06:02 Waiting for dst bucket item count to get to: 50000 curr count: 36660
2021/12/10 01:06:07 Waiting for dst bucket item count to get to: 50000 curr count: 36660
2021/12/10 01:06:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 01:06:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 01:06:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:06:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:32 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/10 01:06:37 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/10 01:06:37 No eventing-con process running
2021/12/10 01:06:37 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/10 01:06:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:06:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:06:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 01:06:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:06:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:06:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:07:01 Waiting for dst bucket item count to get to: 0 curr count: 43980
2021/12/10 01:07:06 Waiting for dst bucket item count to get to: 0 curr count: 43980
2021/12/10 01:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.10s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/10 01:07:16 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:07:16 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 01:07:16 No eventing-con process running
2021/12/10 01:07:16 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/10 01:07:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:07:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:07:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:07:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:07:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:07:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:07:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:07:34 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:07:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/10 01:07:40 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 01:07:45 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 01:07:50 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 01:07:55 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/10 01:07:56 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/10 01:07:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:07:56 Starting bucket ops to source bucket
2021/12/10 01:07:56 Removing node(s): [] from the cluster
2021/12/10 01:08:01 Rebalance progress: 12.5
2021/12/10 01:08:06 Rebalance progress: 12.5
2021/12/10 01:08:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:08:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:08:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:08:17 Rebalance progress: 12.5
2021/12/10 01:08:22 Rebalance progress: 12.5
2021/12/10 01:08:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:08:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:08:32 Rebalance progress: 12.5
2021/12/10 01:08:37 Rebalance progress: 12.5
2021/12/10 01:08:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:08:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:08:47 Waiting for dst bucket item count to get to: 50000 curr count: 37501
2021/12/10 01:08:52 Waiting for dst bucket item count to get to: 50000 curr count: 37501
2021/12/10 01:08:57 Waiting for dst bucket item count to get to: 50000 curr count: 45051
2021/12/10 01:09:02 Waiting for dst bucket item count to get to: 50000 curr count: 45051
2021/12/10 01:09:02 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/10 01:09:02 Post SET ops. Expected item count: 50000 got 50000
2021/12/10 01:09:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:09:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:09:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:09:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:09:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:09:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/10 01:09:27 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/10 01:09:27 No eventing-con process running
2021/12/10 01:09:27 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/10 01:09:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:09:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:09:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/10 01:09:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:09:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:09:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:09:51 Waiting for dst bucket item count to get to: 0 curr count: 33097
2021/12/10 01:09:56 Waiting for dst bucket item count to get to: 0 curr count: 33097
2021/12/10 01:09:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.72s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/10 01:10:06 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:10:06 App: bucket_op_on_update got un-deployed
2021/12/10 01:10:06 No eventing-con process running
2021/12/10 01:10:06 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:10:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:10:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:10:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:10:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:10:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:10:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:10:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:10:24 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:10:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:10:30 Starting bucket ops to source bucket
2021/12/10 01:10:35 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:10:40 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:10:45 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:10:45 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:10:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:10:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:10:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:10:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:10:50 Rebalance progress: 12.5
2021/12/10 01:10:55 Rebalance progress: 12.5
2021/12/10 01:11:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:11:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:11:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:11:06 Rebalance progress: 12.5
2021/12/10 01:11:11 Rebalance progress: 12.5
2021/12/10 01:11:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:11:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 01:11:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:11:22 Rebalance progress: 12.5
2021/12/10 01:11:27 Rebalance progress: 12.5
2021/12/10 01:11:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:11:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:11:37 Rebalance progress: 12.5
2021/12/10 01:11:42 Rebalance progress: 12.5
2021/12/10 01:11:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:11:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:11:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:11:52 Rebalance progress: 12.5
2021/12/10 01:11:57 Rebalance progress: 12.5
2021/12/10 01:12:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:12:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:12:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:12:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 01:12:07 Rebalance progress: 12.5
2021/12/10 01:12:12 Rebalance progress: 12.5
2021/12/10 01:12:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:12:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:12:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:12:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:12:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:12:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:12:47 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:12:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:12:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:12:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:12:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10663
2021/12/10 01:12:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:13:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:13:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:13:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7968
2021/12/10 01:13:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7968
2021/12/10 01:13:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.15s)
=== RUN   TestKVForceFailover
2021/12/10 01:13:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:13:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:13:31 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:13:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:13:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:13:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:13:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:13:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:13:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:13:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:13:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:13:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:13:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:13:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:13:50 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:13:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:13:51 Starting bucket ops to source bucket
2021/12/10 01:13:56 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:14:01 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:14:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:14:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:14:11 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:14:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:14:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:14:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 01:14:31 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 01:14:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/10 01:14:37 Rebalance progress: 12.5
2021/12/10 01:14:42 Rebalance progress: 12.5
2021/12/10 01:14:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:14:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:14:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:14:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/10 01:14:52 Rebalance progress: 12.5
2021/12/10 01:14:57 Rebalance progress: 12.5
2021/12/10 01:15:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:15:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:15:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:15:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:15:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:15:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:15:32 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:15:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:15:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:15:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:15:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7083
2021/12/10 01:15:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7083
2021/12/10 01:15:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:15:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:15:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:16:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5047
2021/12/10 01:16:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5047
2021/12/10 01:16:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (169.97s)
=== RUN   TestKVHardFailoverRecovery
2021/12/10 01:16:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:16:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:16:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:16:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:16:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:16:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:16:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:16:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:16:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:16:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:16:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:16:34 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:16:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:16:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:16:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:16:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:16:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:16:55 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:16:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:16:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:16:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:16:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:17:01 Rebalance progress: 12.5
2021/12/10 01:17:06 Rebalance progress: 12.5
2021/12/10 01:17:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:17:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:17:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:17:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/10 01:18:11 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/10 01:18:16 Rebalance progress: 12.5
2021/12/10 01:18:21 Rebalance progress: 12.5
2021/12/10 01:18:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:18:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:18:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:18:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:18:31 Rebalance progress: 12.5
2021/12/10 01:18:36 Rebalance progress: 12.5
2021/12/10 01:18:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:18:41 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:18:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:18:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:18:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:19:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:19:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:19:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:19:11 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:19:11 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:19:11 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:19:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:19:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:19:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12040
2021/12/10 01:19:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12040
2021/12/10 01:19:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:19:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:19:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:19:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.16s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/10 01:19:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:19:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:19:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:19:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:19:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:19:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:19:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:19:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:19:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:19:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:20:03 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:20:05 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:20:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:20:15 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:20:20 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:20:25 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:20:25 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:20:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:20:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:20:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 01:20:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:20:30 Rebalance progress: 12.5
2021/12/10 01:20:35 Rebalance progress: 12.5
2021/12/10 01:20:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:20:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/10 01:20:46 Rebalance progress: 12.5
2021/12/10 01:20:51 Rebalance progress: 12.5
2021/12/10 01:20:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:20:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:21:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:21:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:21:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:21:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:21:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:21:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4592
2021/12/10 01:21:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:21:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:21:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:21:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.52s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/10 01:21:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:21:54 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:21:54 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:21:54 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:21:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:22:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:22:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:22:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:22:08 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:22:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:22:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:22:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:22:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:22:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:22:29 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:22:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:22:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:22:35 Rebalance progress: 12.5
2021/12/10 01:22:40 Rebalance progress: 12.5
2021/12/10 01:22:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:22:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:22:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:22:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:22:50 Rebalance progress: 12.5
2021/12/10 01:22:55 Rebalance progress: 12.5
2021/12/10 01:23:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:23:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:23:05 Rebalance progress: 12.5
2021/12/10 01:23:10 Rebalance progress: 12.5
2021/12/10 01:23:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:23:15 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:23:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:23:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:23:45 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:23:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:23:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:23:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:23:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/10 01:23:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:23:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:24:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:24:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6088
2021/12/10 01:24:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:24:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:24:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:24:14 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:24:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:24:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:24:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:24:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:24:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:24:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:24:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (163.01s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/10 01:24:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:24:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:24:37 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:24:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:24:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:24:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:24:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:24:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:24:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:24:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:24:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:24:56 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 01:24:56 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:24:56 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 01:24:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:25:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:25:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:25:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:25:09 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:25:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:25:11 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:25:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 01:25:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:25:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:25:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:25:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:25:32 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 01:25:37 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:25:42 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:25:47 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:25:47 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:25:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:25:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:25:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:25:52 Rebalance progress: 12.5
2021/12/10 01:25:57 Rebalance progress: 12.5
2021/12/10 01:26:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:26:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:26:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:26:08 Rebalance progress: 12.5
2021/12/10 01:26:13 Rebalance progress: 12.5
2021/12/10 01:26:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:26:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/10 01:26:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:26:23 Rebalance progress: 12.5
2021/12/10 01:26:28 Rebalance progress: 12.5
2021/12/10 01:26:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:26:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:26:38 Rebalance progress: 12.5
2021/12/10 01:26:43 Rebalance progress: 12.5
2021/12/10 01:26:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:26:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:26:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:26:53 Rebalance progress: 12.5
2021/12/10 01:26:58 Rebalance progress: 12.5
2021/12/10 01:27:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:27:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:27:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/10 01:27:09 Rebalance progress: 12.5
2021/12/10 01:27:14 Rebalance progress: 12.5
2021/12/10 01:27:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:27:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:27:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:27:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:27:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:27:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:27:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:27:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:27:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:27:44 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:27:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:27:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:27:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:27:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12179
2021/12/10 01:27:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:27:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:28:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:28:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12179
2021/12/10 01:28:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:28:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:28:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:28:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:28:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:28:29 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:28:34 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:28:35 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 01:28:35 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:28:35 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 01:28:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:28:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:28:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:28:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:28:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:28:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (260.71s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/10 01:28:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:28:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:28:58 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:28:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:29:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:29:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:29:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:29:17 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 01:29:17 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:29:17 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/10 01:29:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:29:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:30 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:29:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:29:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:29:31 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:29:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/10 01:29:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:29:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:29:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:29:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:29:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/10 01:29:57 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:30:02 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:30:07 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/10 01:30:07 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:30:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:30:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:30:13 Rebalance progress: 12.5
2021/12/10 01:30:18 Rebalance progress: 12.5
2021/12/10 01:30:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:30:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:30:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:30:29 Rebalance progress: 12.5
2021/12/10 01:30:34 Rebalance progress: 12.5
2021/12/10 01:30:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:30:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:30:44 Rebalance progress: 12.5
2021/12/10 01:30:49 Rebalance progress: 12.5
2021/12/10 01:30:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:30:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:30:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:30:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:31:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:31:24 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:31:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:31:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:31:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:31:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7628
2021/12/10 01:31:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:31:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:31:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:31:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3419
2021/12/10 01:31:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3419
2021/12/10 01:31:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:31:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:32:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:06 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:11 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:16 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/10 01:32:26 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/10 01:32:26 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:32:26 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/10 01:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:32:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:32:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:32:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:32:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:32:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (231.14s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/10 01:32:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:32:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:32:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:32:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:32:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:32:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:32:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:32:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:33:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:33:08 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:33:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:33:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:33:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:33:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:33:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:33:29 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:33:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:33:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:33:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:33:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:34:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:34:05 Rebalance progress: 12.5
2021/12/10 01:34:10 Rebalance progress: 12.5
2021/12/10 01:34:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:34:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:34:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:34:20 Rebalance progress: 12.5
2021/12/10 01:34:25 Rebalance progress: 12.5
2021/12/10 01:34:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:34:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:34:35 Rebalance progress: 12.5
2021/12/10 01:34:40 Rebalance progress: 12.5
2021/12/10 01:34:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:34:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:34:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:34:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:34:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:35:16 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:35:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:35:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:35:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:35:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9132
2021/12/10 01:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:35:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:35:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:35:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9132
2021/12/10 01:35:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (180.01s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/10 01:35:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:35:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:35:49 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:35:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/10 01:35:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:35:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:35:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:35:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:36:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:36:08 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:36:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:36:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/10 01:36:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:36:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:36:29 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:36:29 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:36:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:36:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:36:34 Rebalance progress: 12.5
2021/12/10 01:36:39 Rebalance progress: 12.5
2021/12/10 01:36:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:36:45 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:36:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:36:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:37:00 Rebalance progress: 12.5
2021/12/10 01:37:05 Rebalance progress: 12.5
2021/12/10 01:37:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:37:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/10 01:37:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/10 01:37:15 Rebalance progress: 12.5
2021/12/10 01:37:20 Rebalance progress: 12.5
2021/12/10 01:37:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:37:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/10 01:37:30 Rebalance progress: 12.5
2021/12/10 01:37:35 Rebalance progress: 12.5
2021/12/10 01:37:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:37:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:37:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:37:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:37:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:37:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:38:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:38:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:38:06 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:38:06 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:38:06 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:38:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:38:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:38:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8614
2021/12/10 01:38:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:38:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:38:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:38:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8614
2021/12/10 01:38:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (169.87s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/10 01:38:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/10 01:38:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/10 01:38:39 Rebalance progress: 100
2021/12/10 01:38:39 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:39:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:39:09 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/10 01:39:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/10 01:39:15 Removing node(s): [] from the cluster
2021/12/10 01:39:15 Starting bucket ops to source bucket
2021/12/10 01:39:21 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/10 01:39:26 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:39:31 Waiting for app: bucket_op_on_update to get deployed
2021/12/10 01:39:31 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:39:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:39:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/10 01:39:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/10 01:39:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:39:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:39:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:39:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:39:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:40:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/10 01:40:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/10 01:40:01 eventingSetup.go:46: No eventing-con process running
2021/12/10 01:40:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/10 01:40:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:40:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:40:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/10 01:40:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1504
2021/12/10 01:40:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/10 01:40:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/10 01:40:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/10 01:40:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (115.42s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12004.022s

Rebalance tests took 3 hours, 20 minutes and 6 seconds.

Testing Succeeded


Version: versions-09.12.2021-17.56.cfg
Build Log: make-09.12.2021-17.56.log
Server Log: logs-09.12.2021-17.56.tar.gz

Finished