Building

Started building at 2021/12/15 10:28:34
Calculating base
Updating mirror
Basing run on 7.1.0-1913 5c9a76c8a
Updating tree for run 15.12.2021-10.28
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 8cc3b2c, changes since last good build: none
kv_engine is at 1fac684, changes since last good build: none
ns_server is at f99dc7d, changes since last good build: 
 f99dc7d1f MB-50075 Readd code that was inadvertantly removed
 2d301b3d8 Merge remote-tracking branch 'couchbase/cheshire-cat'
 fdb2e2c36 MB-48047, MB-49975: Merge remote-tracking branch 'couchbase/mad-hatter'
 571c8f610 Add can_abort_rebalance to default config
 d6149d774 MB-48047, MB-49975: Update memcached_config on upgrade to 6.6.5
 4b297da3a MB-49702: Revert "MB-18426: Reduce compaction_number_of_kv_workers from 4 to 1"
 4342db3db Give a better error message when unable to set external ports
 898797be7 MB-50028: Fix alternateAddresses breakage for inactiveAdded nodes
couchstore is at 6eea212, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at d49fbde, changes since last good build: 
 d49fbdeb8 code to dispatch jobs based on regex
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 1c6b29cb] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 5a7df29b] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 19795cfa] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 849a1ec6] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 9bf0fe6c] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/6
Total 24 (delta 20), reused 24 (delta 20)
[unstable 927b1296] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/9
Total 56 (delta 47), reused 56 (delta 47)
[unstable 7f28a8e0] MB-46304: Add functionality to disable curl feature
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 193 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 67 (delta 57), reused 67 (delta 57)
[unstable eb664789] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 149 insertions(+), 74 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/4
Total 82 (delta 64), reused 74 (delta 64)
[unstable aca95cf7] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/2
Total 90 (delta 69), reused 80 (delta 69)
[unstable 4a59c623] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/3
Total 5 (delta 2), reused 5 (delta 2)
[unstable 9f8ead1] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/15 10:45:48
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/15 10:47:28 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/15 10:47:39 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 10:47:45 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/15 10:47:45 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 10:47:51 Removing node(s): [] from the cluster
2021/12/15 10:47:56 Rebalance progress: 0
2021/12/15 10:48:01 Rebalance progress: 0.6691261574074074
2021/12/15 10:48:06 Rebalance progress: 5.280671296296297
2021/12/15 10:48:11 Rebalance progress: 9.25925925925926
2021/12/15 10:48:16 Rebalance progress: 13.23784722222222
2021/12/15 10:48:21 Rebalance progress: 17.90364583333333
2021/12/15 10:48:26 Rebalance progress: 21.14076967592592
2021/12/15 10:48:31 Rebalance progress: 25.71614583333333
2021/12/15 10:48:36 Rebalance progress: 100
2021/12/15 10:48:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:48:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:49:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:49:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:49:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:49:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:50:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:50:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:50:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:50:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:51:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:51:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:51:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:51:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:52:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:52:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:52:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:52:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:53:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:53:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:53:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
2021/12/15 10:53:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:0}]
=== RUN   TestAdvancedGetOps
2021/12/15 10:54:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:54:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:54:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:54:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:55:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:55:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:55:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:55:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:56:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:56:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:56:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:56:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:57:11 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:57:26 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:57:41 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:57:56 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/15 10:58:11 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:58:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/15 10:58:18 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 10:58:23 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 10:58:28 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 10:58:33 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/15 10:58:33 App: TestAdvancedGetOps got deployed
2021/12/15 10:58:33 Starting bucket ops to source bucket
2021/12/15 10:58:38 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 10:58:43 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 10:58:43 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 10:58:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 10:58:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 10:58:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 10:58:48 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 10:58:53 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 10:58:58 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 10:59:03 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/15 10:59:03 App: TestAdvancedGetOps got un-deployed
2021/12/15 10:59:03 No eventing-con process running
2021/12/15 10:59:03 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/15 10:59:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 10:59:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 10:59:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 10:59:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 10:59:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 10:59:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (315.09s)
=== RUN   TestAdvancedInsertOps
2021/12/15 10:59:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 10:59:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 10:59:33 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 10:59:38 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 10:59:43 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 10:59:48 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 10:59:48 App: TestAdvancedInsertOps got deployed
2021/12/15 10:59:48 Starting bucket ops to source bucket
2021/12/15 10:59:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 10:59:58 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 11:00:03 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 11:00:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 11:00:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:00:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:00:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:00:09 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:00:14 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:00:19 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:00:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:00:24 App: TestAdvancedInsertOps got un-deployed
2021/12/15 11:00:24 No eventing-con process running
2021/12/15 11:00:24 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 11:00:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:00:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:00:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:00:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:00:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:00:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:00:47 Testing insert operation on source bucket
2021/12/15 11:00:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:00:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:00:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 11:00:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 11:01:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 11:01:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/15 11:01:08 App: TestAdvancedInsertOps got deployed
2021/12/15 11:01:08 Starting bucket ops to source bucket
2021/12/15 11:01:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 11:01:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/15 11:01:19 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 11:01:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:01:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:01:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:01:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:01:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:01:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:01:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/15 11:01:39 App: TestAdvancedInsertOps got un-deployed
2021/12/15 11:01:39 No eventing-con process running
2021/12/15 11:01:39 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/15 11:01:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:01:51 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 11:01:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 11:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:01:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:02:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.47s)
=== RUN   TestAdvancedUpsertOps
2021/12/15 11:02:03 Starting bucket ops to source bucket
2021/12/15 11:02:04 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:02:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:02:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:02:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:02:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:02:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:02:25 App: TestAdvancedUpsertOps got deployed
2021/12/15 11:02:25 Starting bucket ops to source bucket
2021/12/15 11:02:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:02:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 11:02:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:02:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:02:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:02:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:02:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:02:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:02:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:02:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:02:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 11:02:56 No eventing-con process running
2021/12/15 11:02:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 11:02:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:03:08 Waiting for dst bucket item count to get to: 0 curr count: 527
2021/12/15 11:03:13 Waiting for dst bucket item count to get to: 0 curr count: 15
2021/12/15 11:03:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:03:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:03:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:03:19 Testing upsert operation on source bucket
2021/12/15 11:03:19 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:03:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:03:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:03:31 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:03:36 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:03:41 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/15 11:03:41 App: TestAdvancedUpsertOps got deployed
2021/12/15 11:03:41 Starting bucket ops to source bucket
2021/12/15 11:03:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:03:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:03:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:03:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:03:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:03:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:03:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:04:02 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/15 11:04:02 App: TestAdvancedUpsertOps got un-deployed
2021/12/15 11:04:02 No eventing-con process running
2021/12/15 11:04:02 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/15 11:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:04:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:04:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:04:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (132.98s)
=== RUN   TestAdvancedReplaceOps
2021/12/15 11:04:16 Starting bucket ops to source bucket
2021/12/15 11:04:17 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:04:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:04:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:04:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:04:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:04:38 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:04:38 App: TestAdvancedReplaceOps got deployed
2021/12/15 11:04:38 Starting bucket ops to source bucket
2021/12/15 11:04:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:04:49 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 11:04:54 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/15 11:04:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:04:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:04:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:04:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:04:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:05:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:05:09 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:05:14 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:05:14 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 11:05:14 No eventing-con process running
2021/12/15 11:05:14 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 11:05:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:05:26 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:05:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:05:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:05:33 Testing upsert operation on source bucket
2021/12/15 11:05:33 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:05:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:05:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:05:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:05:49 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:05:54 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/15 11:05:54 App: TestAdvancedReplaceOps got deployed
2021/12/15 11:05:54 Starting bucket ops to source bucket
2021/12/15 11:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:05:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:05:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:05:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:06:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:06:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:06:10 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:06:15 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/15 11:06:15 App: TestAdvancedReplaceOps got un-deployed
2021/12/15 11:06:15 No eventing-con process running
2021/12/15 11:06:15 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/15 11:06:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:06:27 Waiting for dst bucket item count to get to: 0 curr count: 345
2021/12/15 11:06:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:06:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:06:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (137.97s)
=== RUN   TestAdvancedDeleteOps
2021/12/15 11:06:34 Starting bucket ops to source bucket
2021/12/15 11:06:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:06:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:06:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:06:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:06:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:06:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:06:55 App: TestAdvancedDeleteOps got deployed
2021/12/15 11:06:55 Starting bucket ops to source bucket
2021/12/15 11:07:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:07:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:07:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:07:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:07:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:07:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:07:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:07:20 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:07:25 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:07:25 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 11:07:25 No eventing-con process running
2021/12/15 11:07:25 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 11:07:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:07:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:07:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:07:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:07:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:07:44 Testing delete operation on source bucket
2021/12/15 11:07:44 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:07:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/15 11:07:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:07:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:08:00 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:08:05 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/15 11:08:05 App: TestAdvancedDeleteOps got deployed
2021/12/15 11:08:05 Starting bucket ops to source bucket
2021/12/15 11:08:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:08:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:08:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:08:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:08:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:08:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:08:20 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:08:25 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/15 11:08:25 App: TestAdvancedDeleteOps got un-deployed
2021/12/15 11:08:25 No eventing-con process running
2021/12/15 11:08:25 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/15 11:08:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:08:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:08:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:08:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.62s)
=== RUN   TestEnoentAdvancedGet
2021/12/15 11:08:39 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:08:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:08:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:08:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:08:56 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:09:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:09:01 App: TestEnoentAdvancedGet got deployed
2021/12/15 11:09:01 Starting bucket ops to source bucket
2021/12/15 11:09:06 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:09:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:09:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:09:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:09:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:09:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:09:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:09:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:09:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:09:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:09:31 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 11:09:31 No eventing-con process running
2021/12/15 11:09:31 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 11:09:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:09:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:09:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:09:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:09:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:10:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:10:04 Testing get enoent operation on source bucket
2021/12/15 11:10:04 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:10:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:10:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:10:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:10:21 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:10:26 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/15 11:10:26 App: TestEnoentAdvancedGet got deployed
2021/12/15 11:10:26 Starting bucket ops to source bucket
2021/12/15 11:10:31 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 11:10:31 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 11:10:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:10:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:10:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:10:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:10:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:10:46 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:10:51 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/15 11:10:51 App: TestEnoentAdvancedGet got un-deployed
2021/12/15 11:10:51 No eventing-con process running
2021/12/15 11:10:51 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/15 11:10:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:11:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 11:11:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:11:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:11:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:11:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.06s)
=== RUN   TestEnoentAdvancedReplace
2021/12/15 11:11:15 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:11:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:11:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:11:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:11:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:11:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:11:36 App: TestEnoentAdvancedReplace got deployed
2021/12/15 11:11:36 Starting bucket ops to source bucket
2021/12/15 11:11:41 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:11:41 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:11:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:11:41 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:11:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:11:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:11:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:11:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:12:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:12:01 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 11:12:01 No eventing-con process running
2021/12/15 11:12:01 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 11:12:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:12:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:12:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:12:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:12:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:12:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:12:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:12:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:12:34 Testing get enoent operation on source bucket
2021/12/15 11:12:35 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:12:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:12:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:12:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:12:51 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:12:56 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/15 11:12:56 App: TestEnoentAdvancedReplace got deployed
2021/12/15 11:12:56 Starting bucket ops to source bucket
2021/12/15 11:13:01 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 11:13:01 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 11:13:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:13:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:13:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:13:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:13:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:13:16 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:13:21 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/15 11:13:21 App: TestEnoentAdvancedReplace got un-deployed
2021/12/15 11:13:21 No eventing-con process running
2021/12/15 11:13:21 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/15 11:13:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:13:33 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 11:13:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:13:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:13:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:13:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.03s)
=== RUN   TestEnoentAdvancedDelete
2021/12/15 11:13:45 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:13:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:13:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:13:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:14:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:14:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:14:06 App: TestEnoentAdvancedDelete got deployed
2021/12/15 11:14:06 Starting bucket ops to source bucket
2021/12/15 11:14:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:14:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:14:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:14:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:14:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:14:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:14:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:14:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:14:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:14:31 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 11:14:31 No eventing-con process running
2021/12/15 11:14:31 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 11:14:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:14:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:14:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:14:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:14:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:14:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:15:04 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:15:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:15:04 Testing get enoent operation on source bucket
2021/12/15 11:15:04 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:15:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:15:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:15:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:15:21 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:15:26 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/15 11:15:26 App: TestEnoentAdvancedDelete got deployed
2021/12/15 11:15:26 Starting bucket ops to source bucket
2021/12/15 11:15:31 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/15 11:15:31 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/15 11:15:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:15:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:15:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:15:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:15:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:15:46 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:15:51 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/15 11:15:51 App: TestEnoentAdvancedDelete got un-deployed
2021/12/15 11:15:51 No eventing-con process running
2021/12/15 11:15:51 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/15 11:15:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:16:03 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 11:16:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:16:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:16:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:16:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.03s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/15 11:16:15 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:16:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 11:16:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:16:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:16:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:16:36 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:16:36 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 11:16:36 Starting bucket ops to source bucket
2021/12/15 11:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:16:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:16:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:16:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:16:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:16:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:16:51 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:16:56 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:16:56 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 11:16:56 No eventing-con process running
2021/12/15 11:16:56 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 11:16:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:17:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:17:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:17:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:17:10 Pumping document in the destination bucket
2021/12/15 11:17:10 Starting bucket ops to source bucket
2021/12/15 11:17:10 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:17:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/15 11:17:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:17:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:17:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:17:31 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/15 11:17:31 App: TestAdvancedInsertKeyExist got deployed
2021/12/15 11:17:31 Starting bucket ops to source bucket
2021/12/15 11:17:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:17:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:17:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:17:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:17:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:17:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:17:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:17:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:17:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:18:02 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/15 11:18:02 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/15 11:18:02 No eventing-con process running
2021/12/15 11:18:02 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/15 11:18:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:18:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:18:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:18:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:18:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:18:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.75s)
=== RUN   TestExipryGet
2021/12/15 11:18:25 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:18:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:18:32 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:18:37 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:18:42 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:18:47 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:18:47 App: TestExipryGet got deployed
2021/12/15 11:18:47 Starting bucket ops to source bucket
2021/12/15 11:19:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:19:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:19:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:19:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:19:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:19:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:19:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:19:47 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:19:52 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:19:52 App: TestExipryGet got un-deployed
2021/12/15 11:19:52 No eventing-con process running
2021/12/15 11:19:52 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 11:19:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:19:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:20:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:20:06 Pumping document in the destination bucket for expiry get
2021/12/15 11:20:06 Starting bucket ops to source bucket
2021/12/15 11:20:06 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:20:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:20:12 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:20:17 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:20:22 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:20:27 Waiting for app: TestExipryGet to get deployed
2021/12/15 11:20:27 App: TestExipryGet got deployed
2021/12/15 11:20:27 Starting bucket ops to source bucket
2021/12/15 11:21:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:21:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:21:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:21:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:21:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:21:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:21:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:21:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:21:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:21:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/15 11:21:43 App: TestExipryGet got un-deployed
2021/12/15 11:21:43 No eventing-con process running
2021/12/15 11:21:43 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/15 11:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:21:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:21:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:22:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (216.00s)
=== RUN   TestExipryInsert
2021/12/15 11:22:01 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:22:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 11:22:08 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:22:13 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:22:18 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:22:23 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:22:23 App: TestExipryInsert got deployed
2021/12/15 11:22:23 Starting bucket ops to source bucket
2021/12/15 11:23:08 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 11:23:13 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/15 11:23:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:23:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:23:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:23:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:23:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:23:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:23:28 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:23:33 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:23:33 App: TestExipryInsert got un-deployed
2021/12/15 11:23:33 No eventing-con process running
2021/12/15 11:23:33 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 11:23:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:23:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:23:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:23:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:23:47 Testing on the destination bucket for expiry insert
2021/12/15 11:23:47 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:23:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/15 11:23:53 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:23:58 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:24:03 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:24:08 Waiting for app: TestExipryInsert to get deployed
2021/12/15 11:24:08 App: TestExipryInsert got deployed
2021/12/15 11:24:08 Starting bucket ops to source bucket
2021/12/15 11:24:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:24:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:24:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:24:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:24:58 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:24:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:25:04 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:25:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:25:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:25:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/15 11:25:19 App: TestExipryInsert got un-deployed
2021/12/15 11:25:19 No eventing-con process running
2021/12/15 11:25:19 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/15 11:25:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:25:30 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/15 11:25:35 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/15 11:25:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:25:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:25:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.78s)
=== RUN   TestExipryReplace
2021/12/15 11:25:42 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:25:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:25:48 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:25:53 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:25:58 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:26:03 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:26:03 App: TestExipryReplace got deployed
2021/12/15 11:26:03 Starting bucket ops to source bucket
2021/12/15 11:26:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:26:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:26:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:26:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:26:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:26:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:26:59 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:27:04 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:27:09 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:27:14 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:27:14 App: TestExipryReplace got un-deployed
2021/12/15 11:27:14 No eventing-con process running
2021/12/15 11:27:14 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 11:27:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:27:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:27:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:27:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:27:28 Testing on the destination bucket for expiry insert
2021/12/15 11:27:28 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:27:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:27:34 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:27:39 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:27:44 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:27:49 Waiting for app: TestExipryReplace to get deployed
2021/12/15 11:27:49 App: TestExipryReplace got deployed
2021/12/15 11:27:49 Starting bucket ops to source bucket
2021/12/15 11:28:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:28:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:28:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:28:34 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:28:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:28:39 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:28:44 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:28:49 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:28:54 Waiting for app: TestExipryReplace to get un-deployed
2021/12/15 11:28:54 App: TestExipryReplace got un-deployed
2021/12/15 11:28:54 No eventing-con process running
2021/12/15 11:28:54 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/15 11:28:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:29:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:29:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:29:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.64s)
=== RUN   TestExipryUpsert
2021/12/15 11:29:13 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:29:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 11:29:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:29:24 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:29:29 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:29:34 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:29:34 App: TestExipryUpsert got deployed
2021/12/15 11:29:34 Starting bucket ops to source bucket
2021/12/15 11:30:19 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:30:24 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:30:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:30:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:30:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:30:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:30:29 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:30:34 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:30:39 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:30:44 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:30:44 App: TestExipryUpsert got un-deployed
2021/12/15 11:30:44 No eventing-con process running
2021/12/15 11:30:44 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 11:30:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:30:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:30:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:30:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:30:58 Testing on the destination bucket for expiry insert
2021/12/15 11:30:58 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:30:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/15 11:31:04 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:31:09 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:31:14 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:31:19 Waiting for app: TestExipryUpsert to get deployed
2021/12/15 11:31:19 App: TestExipryUpsert got deployed
2021/12/15 11:31:19 Starting bucket ops to source bucket
2021/12/15 11:32:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:32:10 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:32:15 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:32:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:32:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:32:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:32:20 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:32:25 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:32:30 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:32:35 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/15 11:32:35 App: TestExipryUpsert got un-deployed
2021/12/15 11:32:35 No eventing-con process running
2021/12/15 11:32:35 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/15 11:32:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:32:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:32:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:32:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:32:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (220.65s)
=== RUN   TestCasReplace
2021/12/15 11:32:53 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:32:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:33:00 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:33:05 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:33:10 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:33:15 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:33:15 App: TestCasReplace got deployed
2021/12/15 11:33:15 Starting bucket ops to source bucket
2021/12/15 11:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:33:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:33:15 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:33:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:33:20 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:33:25 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:33:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:33:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:33:35 App: TestCasReplace got un-deployed
2021/12/15 11:33:35 No eventing-con process running
2021/12/15 11:33:35 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 11:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:33:47 Waiting for dst bucket item count to get to: 0 curr count: 21
2021/12/15 11:33:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:33:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:33:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:33:54 Testing on the destination bucket for upsert with cas
2021/12/15 11:33:54 Starting bucket ops to source bucket
2021/12/15 11:33:54 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:33:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/15 11:34:00 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:34:05 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:34:10 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:34:15 Waiting for app: TestCasReplace to get deployed
2021/12/15 11:34:15 App: TestCasReplace got deployed
2021/12/15 11:34:15 Starting bucket ops to source bucket
2021/12/15 11:34:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:34:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:34:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:34:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:34:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:34:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:34:30 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:34:35 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:34:40 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:34:45 Waiting for app: TestCasReplace to get un-deployed
2021/12/15 11:34:45 App: TestCasReplace got un-deployed
2021/12/15 11:34:45 No eventing-con process running
2021/12/15 11:34:45 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/15 11:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:34:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:34:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:35:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.28s)
=== RUN   TestCasDelete
2021/12/15 11:35:04 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:35:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:35:10 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:35:15 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:35:20 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:35:25 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:35:25 App: TestCasDelete got deployed
2021/12/15 11:35:25 Starting bucket ops to source bucket
2021/12/15 11:35:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:35:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:35:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:35:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:35:30 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:35:35 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:35:40 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:35:45 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:35:45 App: TestCasDelete got un-deployed
2021/12/15 11:35:45 No eventing-con process running
2021/12/15 11:35:45 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 11:35:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:35:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:35:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:35:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:35:59 Testing on the destination bucket for delete with cas
2021/12/15 11:35:59 Starting bucket ops to source bucket
2021/12/15 11:36:00 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:36:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:36:06 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:36:11 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:36:16 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:36:21 Waiting for app: TestCasDelete to get deployed
2021/12/15 11:36:21 App: TestCasDelete got deployed
2021/12/15 11:36:21 Starting bucket ops to source bucket
2021/12/15 11:36:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:36:31 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:36:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:36:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:36:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 11:36:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:36:36 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:36:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:36:46 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:36:51 Waiting for app: TestCasDelete to get un-deployed
2021/12/15 11:36:51 App: TestCasDelete got un-deployed
2021/12/15 11:36:51 No eventing-con process running
2021/12/15 11:36:51 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/15 11:36:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:37:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:37:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:37:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:37:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.84s)
=== RUN   TestCountersIncrement
2021/12/15 11:37:15 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 11:37:20 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 11:37:21 Removing node(s): [] from the cluster
2021/12/15 11:37:26 Rebalance progress: 100
2021/12/15 11:37:26 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:37:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 11:37:32 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:37:37 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:37:42 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:37:47 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:37:47 App: TestCountersIncrement got deployed
2021/12/15 11:37:47 Starting bucket ops to source bucket
2021/12/15 11:37:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 11:37:58 Waiting for dst bucket item count to get to: 1024 curr count: 566
2021/12/15 11:38:03 Waiting for dst bucket item count to get to: 1024 curr count: 566
2021/12/15 11:38:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 11:38:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:38:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:38:08 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:38:13 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:38:18 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:38:23 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:38:23 App: TestCountersIncrement got un-deployed
2021/12/15 11:38:23 No eventing-con process running
2021/12/15 11:38:23 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 11:38:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:38:35 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/15 11:38:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:38:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:38:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:38:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:38:47 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:38:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/15 11:38:53 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:38:58 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:39:03 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:39:08 Waiting for app: TestCountersIncrement to get deployed
2021/12/15 11:39:08 App: TestCountersIncrement got deployed
2021/12/15 11:39:08 Starting bucket ops to source bucket
2021/12/15 11:39:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 11:39:19 Waiting for dst bucket item count to get to: 2048 curr count: 1022
2021/12/15 11:39:24 Waiting for dst bucket item count to get to: 2048 curr count: 2046
2021/12/15 11:39:24 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 11:39:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:39:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:39:29 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:39:34 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:39:39 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:39:44 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/15 11:39:44 App: TestCountersIncrement got un-deployed
2021/12/15 11:39:44 No eventing-con process running
2021/12/15 11:39:45 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/15 11:39:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:39:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 11:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:39:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:40:08 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 11:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:40:08 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 11:40:13 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.57s)
=== RUN   TestCountersDecrement
2021/12/15 11:40:13 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 11:40:13 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 11:40:14 Removing node(s): [] from the cluster
2021/12/15 11:40:19 Rebalance progress: 100
2021/12/15 11:40:19 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:40:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 11:40:25 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:40:30 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:40:35 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:40:40 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:40:40 App: TestCountersDecrement got deployed
2021/12/15 11:40:40 Starting bucket ops to source bucket
2021/12/15 11:40:46 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 11:40:51 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/15 11:40:56 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/15 11:40:56 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/15 11:40:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:40:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:41:01 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:41:06 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:41:11 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:41:16 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:41:16 App: TestCountersDecrement got un-deployed
2021/12/15 11:41:16 No eventing-con process running
2021/12/15 11:41:16 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 11:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:41:28 Waiting for dst bucket item count to get to: 0 curr count: 635
2021/12/15 11:41:33 Waiting for dst bucket item count to get to: 0 curr count: 122
2021/12/15 11:41:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:41:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:41:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:41:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:41:45 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:41:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/15 11:41:51 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:41:56 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:42:01 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:42:06 Waiting for app: TestCountersDecrement to get deployed
2021/12/15 11:42:06 App: TestCountersDecrement got deployed
2021/12/15 11:42:06 Starting bucket ops to source bucket
2021/12/15 11:42:12 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/15 11:42:12 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/15 11:42:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:42:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:42:17 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:42:22 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:42:27 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:42:32 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/15 11:42:32 App: TestCountersDecrement got un-deployed
2021/12/15 11:42:32 No eventing-con process running
2021/12/15 11:42:32 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/15 11:42:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:42:44 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/15 11:42:49 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/15 11:42:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:42:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:43:01 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 11:43:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 11:43:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:43:06 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/15 11:43:11 Rebalance progress: 100
--- PASS: TestCountersDecrement (177.84s)
=== RUN   TestCacheGet
2021/12/15 11:43:11 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:43:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:43:17 Waiting for app: TestCacheGet to get deployed
2021/12/15 11:43:22 Waiting for app: TestCacheGet to get deployed
2021/12/15 11:43:27 Waiting for app: TestCacheGet to get deployed
2021/12/15 11:43:32 Waiting for app: TestCacheGet to get deployed
2021/12/15 11:43:32 App: TestCacheGet got deployed
2021/12/15 11:43:32 Starting bucket ops to source bucket
2021/12/15 11:43:37 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 11:43:42 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/15 11:43:47 Waiting for dst bucket item count to get to: 20 curr count: 1
2021/12/15 11:43:52 Waiting for dst bucket item count to get to: 20 curr count: 1
2021/12/15 11:43:57 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/15 11:44:02 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/15 11:44:02 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/15 11:44:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:44:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:44:08 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 11:44:13 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 11:44:18 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 11:44:23 Waiting for app: TestCacheGet to get un-deployed
2021/12/15 11:44:23 App: TestCacheGet got un-deployed
2021/12/15 11:44:23 No eventing-con process running
2021/12/15 11:44:23 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/15 11:44:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:44:34 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/15 11:44:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:44:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:44:46 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/15 11:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.21s)
=== RUN   TestCacheRYOW
2021/12/15 11:44:46 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:44:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/15 11:44:52 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 11:44:57 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 11:45:02 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 11:45:07 Waiting for app: TestCacheRYOW to get deployed
2021/12/15 11:45:07 App: TestCacheRYOW got deployed
2021/12/15 11:45:07 Starting bucket ops to source bucket
2021/12/15 11:45:13 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/15 11:45:18 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/15 11:45:23 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/15 11:45:23 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/15 11:45:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:45:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:45:28 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 11:45:33 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 11:45:38 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 11:45:43 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/15 11:45:43 App: TestCacheRYOW got un-deployed
2021/12/15 11:45:43 No eventing-con process running
2021/12/15 11:45:43 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/15 11:45:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:45:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:45:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:46:06 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.18s)
=== RUN   TestCacheExpiry
2021/12/15 11:46:06 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:46:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/15 11:46:13 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 11:46:18 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 11:46:23 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 11:46:28 Waiting for app: TestCacheExpiry to get deployed
2021/12/15 11:46:28 App: TestCacheExpiry got deployed
2021/12/15 11:46:28 Starting bucket ops to source bucket
2021/12/15 11:46:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:46:38 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/15 11:46:43 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/15 11:46:48 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/15 11:46:53 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/15 11:46:53 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:46:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:46:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:46:58 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 11:47:03 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 11:47:08 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 11:47:13 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/15 11:47:13 App: TestCacheExpiry got un-deployed
2021/12/15 11:47:13 No eventing-con process running
2021/12/15 11:47:13 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/15 11:47:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:47:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:47:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:47:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:47:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (90.12s)
=== RUN   TestCacheExpiryCustom
2021/12/15 11:47:36 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:47:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 11:47:43 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 11:47:48 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 11:47:53 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 11:47:58 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/15 11:47:58 App: TestCacheExpiryCustom got deployed
2021/12/15 11:47:58 Starting bucket ops to source bucket
2021/12/15 11:48:03 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:48:08 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/15 11:48:13 Waiting for dst bucket item count to get to: 100 curr count: 3
2021/12/15 11:48:18 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/15 11:48:23 Waiting for dst bucket item count to get to: 100 curr count: 86
2021/12/15 11:48:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:48:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:48:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:48:28 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 11:48:33 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 11:48:38 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 11:48:43 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/15 11:48:43 App: TestCacheExpiryCustom got un-deployed
2021/12/15 11:48:43 No eventing-con process running
2021/12/15 11:48:43 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/15 11:48:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:48:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/15 11:48:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:48:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:49:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:49:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (90.27s)
=== RUN   TestCacheOverflowCustom
2021/12/15 11:49:07 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:49:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/15 11:49:13 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 11:49:18 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 11:49:23 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 11:49:28 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/15 11:49:28 App: TestCacheOverflowCustom got deployed
2021/12/15 11:49:28 Starting bucket ops to source bucket
2021/12/15 11:49:33 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/15 11:49:38 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/15 11:49:43 Waiting for dst bucket item count to get to: 35 curr count: 3
2021/12/15 11:49:48 Waiting for dst bucket item count to get to: 35 curr count: 24
2021/12/15 11:49:53 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/15 11:49:58 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/15 11:50:03 Waiting for dst bucket item count to get to: 35 curr count: 34
2021/12/15 11:50:03 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/15 11:50:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:50:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:50:08 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 11:50:13 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 11:50:18 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 11:50:23 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/15 11:50:23 App: TestCacheOverflowCustom got un-deployed
2021/12/15 11:50:23 No eventing-con process running
2021/12/15 11:50:23 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/15 11:50:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:50:35 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/15 11:50:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:50:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:50:47 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/15 11:50:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.21s)
=== RUN   TestStrictMode
2021/12/15 11:50:47 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:50:48 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.05s)
=== RUN   TestDataTypes
2021/12/15 11:50:48 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:50:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/15 11:50:54 Waiting for app: TestDataTypes to get deployed
2021/12/15 11:50:59 Waiting for app: TestDataTypes to get deployed
2021/12/15 11:51:04 Waiting for app: TestDataTypes to get deployed
2021/12/15 11:51:09 Waiting for app: TestDataTypes to get deployed
2021/12/15 11:51:09 App: TestDataTypes got deployed
2021/12/15 11:51:09 Starting bucket ops to source bucket
2021/12/15 11:51:14 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/15 11:51:19 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/15 11:51:19 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/15 11:51:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:51:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:51:24 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 11:51:29 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 11:51:34 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 11:51:39 Waiting for app: TestDataTypes to get un-deployed
2021/12/15 11:51:39 App: TestDataTypes got un-deployed
2021/12/15 11:51:39 No eventing-con process running
2021/12/15 11:51:39 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/15 11:51:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:51:51 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 11:51:56 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 11:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:51:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:52:08 Waiting for dst bucket item count to get to: 0 curr count: 12
2021/12/15 11:52:13 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/15 11:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (85.10s)
=== RUN   TestEnoentGet
2021/12/15 11:52:13 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:52:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:52:19 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:52:24 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:52:29 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:52:34 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:52:34 App: TestEnoentGet got deployed
2021/12/15 11:52:34 Starting bucket ops to source bucket
2021/12/15 11:52:39 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:52:44 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/15 11:52:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:52:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:52:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:52:50 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:52:55 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:53:00 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:53:05 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:53:05 App: TestEnoentGet got un-deployed
2021/12/15 11:53:05 No eventing-con process running
2021/12/15 11:53:05 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 11:53:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:53:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:53:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:53:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:53:28 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/15 11:53:33 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/15 11:53:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:53:33 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:53:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/15 11:53:39 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:53:44 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:53:49 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:53:54 Waiting for app: TestEnoentGet to get deployed
2021/12/15 11:53:54 App: TestEnoentGet got deployed
2021/12/15 11:53:54 Starting bucket ops to source bucket
2021/12/15 11:53:59 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:54:04 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/15 11:54:05 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:54:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:54:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:54:10 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:54:15 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:54:20 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:54:25 Waiting for app: TestEnoentGet to get un-deployed
2021/12/15 11:54:25 App: TestEnoentGet got un-deployed
2021/12/15 11:54:25 No eventing-con process running
2021/12/15 11:54:25 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/15 11:54:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:54:36 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:54:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:54:48 Waiting for dst bucket item count to get to: 0 curr count: 78
2021/12/15 11:54:53 Waiting for dst bucket item count to get to: 0 curr count: 28
2021/12/15 11:54:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.25s)
=== RUN   TestEnoentDelete
2021/12/15 11:54:53 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:54:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:54:59 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:55:04 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:55:09 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:55:14 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:55:14 App: TestEnoentDelete got deployed
2021/12/15 11:55:14 Starting bucket ops to source bucket
2021/12/15 11:55:20 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:55:25 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/15 11:55:25 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:55:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:55:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:55:30 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:55:35 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:55:40 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:55:45 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:55:45 App: TestEnoentDelete got un-deployed
2021/12/15 11:55:45 No eventing-con process running
2021/12/15 11:55:45 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 11:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:55:57 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:55:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:55:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:56:08 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/15 11:56:13 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/15 11:56:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:56:13 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:56:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 11:56:20 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:56:25 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:56:30 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:56:35 Waiting for app: TestEnoentDelete to get deployed
2021/12/15 11:56:35 App: TestEnoentDelete got deployed
2021/12/15 11:56:35 Starting bucket ops to source bucket
2021/12/15 11:56:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 11:56:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/15 11:56:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 11:56:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:56:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:56:50 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:56:55 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:57:00 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:57:05 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/15 11:57:05 App: TestEnoentDelete got un-deployed
2021/12/15 11:57:05 No eventing-con process running
2021/12/15 11:57:05 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/15 11:57:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:57:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 11:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:57:29 Waiting for dst bucket item count to get to: 0 curr count: 83
2021/12/15 11:57:34 Waiting for dst bucket item count to get to: 0 curr count: 33
2021/12/15 11:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.23s)
=== RUN   TestError
2021/12/15 11:57:44 Waiting for app: TestError to get un-deployed
2021/12/15 11:57:44 App: TestError got un-deployed
2021/12/15 11:57:44 No eventing-con process running
2021/12/15 11:57:44 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/15 11:57:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:57:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:57:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:57:57 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:57:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/15 11:58:03 Waiting for app: TestError to get deployed
2021/12/15 11:58:08 Waiting for app: TestError to get deployed
2021/12/15 11:58:13 Waiting for app: TestError to get deployed
2021/12/15 11:58:18 Waiting for app: TestError to get deployed
2021/12/15 11:58:18 App: TestError got deployed
2021/12/15 11:58:18 Starting bucket ops to source bucket
2021/12/15 11:58:24 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/15 11:58:29 Waiting for dst bucket item count to get to: 300 curr count: 144
2021/12/15 11:58:29 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/15 11:58:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 11:58:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 11:58:34 Waiting for app: TestError to get un-deployed
2021/12/15 11:58:39 Waiting for app: TestError to get un-deployed
2021/12/15 11:58:44 Waiting for app: TestError to get un-deployed
2021/12/15 11:58:49 Waiting for app: TestError to get un-deployed
2021/12/15 11:58:49 App: TestError got un-deployed
2021/12/15 11:58:49 No eventing-con process running
2021/12/15 11:58:49 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/15 11:58:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:59:01 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/15 11:59:06 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/15 11:59:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:59:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:59:17 Waiting for dst bucket item count to get to: 0 curr count: 144
2021/12/15 11:59:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (103.92s)
=== RUN   TestCRLF
2021/12/15 11:59:27 Waiting for app: TestCRLF to get un-deployed
2021/12/15 11:59:27 App: TestCRLF got un-deployed
2021/12/15 11:59:27 No eventing-con process running
2021/12/15 11:59:27 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/15 11:59:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:59:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:59:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 11:59:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 11:59:41 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/15 11:59:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/15 11:59:47 Waiting for app: TestCRLF to get deployed
2021/12/15 11:59:52 Waiting for app: TestCRLF to get deployed
2021/12/15 11:59:57 Waiting for app: TestCRLF to get deployed
2021/12/15 12:00:02 Waiting for app: TestCRLF to get deployed
2021/12/15 12:00:02 App: TestCRLF got deployed
2021/12/15 12:00:02 Starting bucket ops to source bucket
2021/12/15 12:00:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 12:00:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 12:00:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 12:00:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:00:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:00:18 Waiting for app: TestCRLF to get un-deployed
2021/12/15 12:00:23 Waiting for app: TestCRLF to get un-deployed
2021/12/15 12:00:28 Waiting for app: TestCRLF to get un-deployed
2021/12/15 12:00:33 Waiting for app: TestCRLF to get un-deployed
2021/12/15 12:00:33 App: TestCRLF got un-deployed
2021/12/15 12:00:33 No eventing-con process running
2021/12/15 12:00:33 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/15 12:00:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:00:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 12:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:00:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:00:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 12:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (98.78s)
=== RUN   TestImportExport
2021/12/15 12:01:06 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:01:06 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 12:01:06 No eventing-con process running
2021/12/15 12:01:06 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/15 12:01:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:01:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:01:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:01:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:01:25 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:01:25 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 12:01:25 No eventing-con process running
2021/12/15 12:01:25 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/15 12:01:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:01:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:01:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:01:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:01:38 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:01:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 12:01:40 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:01:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 12:01:46 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 12:01:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 12:01:56 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 12:02:01 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/15 12:02:01 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/15 12:02:06 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 12:02:11 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 12:02:16 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/15 12:02:16 App: TestImportExport_bucket_op_on_update got deployed
2021/12/15 12:02:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:02:21 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:02:26 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:02:31 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:02:36 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:02:36 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 12:02:36 No eventing-con process running
2021/12/15 12:02:36 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 12:02:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:02:41 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:02:46 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:02:51 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:02:56 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:02:56 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 12:02:56 No eventing-con process running
2021/12/15 12:02:56 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 12:03:14 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/15 12:03:14 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/15 12:03:14 No eventing-con process running
2021/12/15 12:03:14 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/15 12:03:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:03:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:03:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:03:32 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/15 12:03:32 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/15 12:03:32 No eventing-con process running
2021/12/15 12:03:32 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/15 12:03:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:03:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:03:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.62s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/15 12:03:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:03:56 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:03:56 No eventing-con process running
2021/12/15 12:03:56 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/15 12:03:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:04:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:04:09 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:04:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:04:11 Starting bucket ops to source bucket
2021/12/15 12:04:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:04:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:04:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:04:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:04:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:04:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 12:04:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 12:04:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:04:39 Undeploying app: bucket_op_on_update
2021/12/15 12:04:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:04:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:04:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:04:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:04:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:04:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:04:59 No eventing-con process running
2021/12/15 12:05:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:05:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:05:43 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:05:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:05:44 Starting bucket ops to source bucket
2021/12/15 12:05:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:05:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:06:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:06:07 Waiting for dst bucket item count to get to: 5000 curr count: 1747
2021/12/15 12:06:12 Waiting for dst bucket item count to get to: 5000 curr count: 4247
2021/12/15 12:06:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:06:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 12:06:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 12:06:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:06:17 Undeploying app: bucket_op_on_update
2021/12/15 12:06:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:06:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:06:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:06:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:06:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:06:37 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:06:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:06:37 No eventing-con process running
2021/12/15 12:06:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:06:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:07:21 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:07:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:07:22 Starting bucket ops to source bucket
2021/12/15 12:07:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:07:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:07:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:07:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:07:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:07:50 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 12:07:50 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 12:07:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:07:50 Undeploying app: bucket_op_on_update
2021/12/15 12:07:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:07:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:08:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:08:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:08:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:08:11 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:08:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:08:11 No eventing-con process running
2021/12/15 12:08:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:08:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:08:55 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:08:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:08:56 Starting bucket ops to source bucket
2021/12/15 12:09:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:09:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:09:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:09:19 Waiting for dst bucket item count to get to: 5000 curr count: 2736
2021/12/15 12:09:24 Waiting for dst bucket item count to get to: 5000 curr count: 2736
2021/12/15 12:09:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:09:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 12:09:29 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 12:09:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:09:29 Undeploying app: bucket_op_on_update
2021/12/15 12:09:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:09:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:09:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:09:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:09:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:09:49 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:09:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:09:49 No eventing-con process running
2021/12/15 12:09:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:09:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:10:33 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:10:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:10:34 Starting bucket ops to source bucket
2021/12/15 12:10:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:10:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:10:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:10:57 Waiting for dst bucket item count to get to: 5000 curr count: 1399
2021/12/15 12:11:02 Waiting for dst bucket item count to get to: 5000 curr count: 3899
2021/12/15 12:11:02 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:11:07 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/15 12:11:07 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/15 12:11:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:11:07 Undeploying app: bucket_op_on_update
2021/12/15 12:11:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:11:12 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:11:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:11:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:11:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/15 12:11:27 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/15 12:11:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:11:27 No eventing-con process running
2021/12/15 12:11:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:11:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:12:11 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (505.52s)
=== RUN   TestOnUpdateN1QLOp
2021/12/15 12:12:21 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 12:12:21 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 12:12:21 No eventing-con process running
2021/12/15 12:12:21 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/15 12:12:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:12:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:12:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:12:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:12:35 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:12:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/15 12:12:36 Starting bucket ops to source bucket
2021/12/15 12:12:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:12:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:12:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:12:59 Waiting for dst bucket item count to get to: 5000 curr count: 669
2021/12/15 12:13:04 Waiting for dst bucket item count to get to: 5000 curr count: 669
2021/12/15 12:13:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:13:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:13:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:13:09 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 12:13:14 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 12:13:19 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 12:13:24 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/15 12:13:24 App: TestOnUpdateN1QLOp got un-deployed
2021/12/15 12:13:24 No eventing-con process running
2021/12/15 12:13:24 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/15 12:13:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:13:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:13:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:13:48 Waiting for dst bucket item count to get to: 0 curr count: 3091
2021/12/15 12:13:53 Waiting for dst bucket item count to get to: 0 curr count: 564
2021/12/15 12:13:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.42s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/15 12:14:03 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 12:14:03 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 12:14:03 No eventing-con process running
2021/12/15 12:14:03 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/15 12:14:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:14:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:14:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:14:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:14:16 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:14:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:14:18 Starting bucket ops to source bucket
2021/12/15 12:14:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:14:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:14:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:14:41 Waiting for dst bucket item count to get to: 5000 curr count: 724
2021/12/15 12:14:46 Waiting for dst bucket item count to get to: 5000 curr count: 724
2021/12/15 12:14:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:14:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:14:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:14:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 12:14:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 12:15:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 12:15:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/15 12:15:06 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/15 12:15:06 No eventing-con process running
2021/12/15 12:15:06 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/15 12:15:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:15:18 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:15:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:15:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:15:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:15:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:15:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.57s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/15 12:15:44 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 12:15:44 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 12:15:44 No eventing-con process running
2021/12/15 12:15:44 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 12:15:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:15:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:15:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:15:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:15:58 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:15:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:15:59 Starting bucket ops to source bucket
2021/12/15 12:16:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:16:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:16:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:16:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:16:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 12:16:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:16:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:16:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:16:32 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 12:16:37 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 12:16:42 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 12:16:47 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 12:16:47 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 12:16:47 No eventing-con process running
2021/12/15 12:16:47 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 12:16:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:16:59 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:17:04 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:17:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:17:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:17:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.52s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/15 12:17:26 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 12:17:26 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 12:17:26 No eventing-con process running
2021/12/15 12:17:26 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/15 12:17:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:17:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:17:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:17:39 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:17:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/15 12:17:41 Starting bucket ops to source bucket
2021/12/15 12:17:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:17:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:17:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:18:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:18:04 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:18:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:18:09 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 12:18:14 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 12:18:19 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 12:18:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/15 12:18:24 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/15 12:18:24 No eventing-con process running
2021/12/15 12:18:24 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/15 12:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:18:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:18:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:18:48 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:18:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (91.67s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/15 12:18:58 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 12:18:58 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 12:18:58 No eventing-con process running
2021/12/15 12:18:58 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/15 12:18:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:19:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:19:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:19:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:19:11 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:19:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/15 12:19:13 Starting bucket ops to source bucket
2021/12/15 12:19:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:19:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:19:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:19:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:19:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:19:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:19:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:19:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 12:19:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 12:19:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 12:19:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/15 12:19:56 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/15 12:19:56 No eventing-con process running
2021/12/15 12:19:56 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/15 12:19:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:20:08 Waiting for dst bucket item count to get to: 0 curr count: 3299
2021/12/15 12:20:13 Waiting for dst bucket item count to get to: 0 curr count: 799
2021/12/15 12:20:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:20:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:20:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:20:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.19s)
=== RUN   TestOnDeleteBucketOp
2021/12/15 12:20:35 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 12:20:35 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 12:20:35 No eventing-con process running
2021/12/15 12:20:35 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/15 12:20:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:20:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:20:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:20:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:20:48 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:20:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 12:20:50 Starting bucket ops to source bucket
2021/12/15 12:21:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:21:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:21:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:21:15 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/15 12:21:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:21:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:21:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:21:20 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 12:21:25 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 12:21:30 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 12:21:35 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/15 12:21:35 App: TestOnDeleteBucketOp got un-deployed
2021/12/15 12:21:36 No eventing-con process running
2021/12/15 12:21:36 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/15 12:21:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:21:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:21:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (89.85s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/15 12:22:05 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 12:22:05 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 12:22:05 No eventing-con process running
2021/12/15 12:22:05 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/15 12:22:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:22:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:22:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:22:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:22:18 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:22:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/15 12:22:19 Starting bucket ops to source bucket
2021/12/15 12:22:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:22:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:22:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:22:45 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/15 12:22:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:22:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:22:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:22:50 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 12:22:55 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 12:23:00 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 12:23:05 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/15 12:23:05 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/15 12:23:05 No eventing-con process running
2021/12/15 12:23:05 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/15 12:23:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:23:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:23:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:23:24 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:23:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.85s)
=== RUN   TestTimerBucketOp
2021/12/15 12:23:34 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 12:23:34 App: TestTimerBucketOp got un-deployed
2021/12/15 12:23:34 No eventing-con process running
2021/12/15 12:23:34 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/15 12:23:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:23:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:23:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:23:48 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:23:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 12:23:49 Starting bucket ops to source bucket
2021/12/15 12:23:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:24:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:24:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:24:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:24:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:24:22 Waiting for dst bucket item count to get to: 5000 curr count: 573
2021/12/15 12:24:27 Waiting for dst bucket item count to get to: 5000 curr count: 2925
2021/12/15 12:24:32 Waiting for dst bucket item count to get to: 5000 curr count: 4685
2021/12/15 12:24:37 Waiting for dst bucket item count to get to: 5000 curr count: 4833
2021/12/15 12:24:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:24:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:24:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:24:42 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 12:24:47 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 12:24:53 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 12:24:58 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/15 12:24:58 App: TestTimerBucketOp got un-deployed
2021/12/15 12:24:58 No eventing-con process running
2021/12/15 12:24:58 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/15 12:24:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:25:09 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:25:14 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 12:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:25:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:25:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:25:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.69s)
=== RUN   TestTimerInPastBucketOp
2021/12/15 12:25:36 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 12:25:36 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 12:25:36 No eventing-con process running
2021/12/15 12:25:36 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/15 12:25:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:25:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:25:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:25:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:25:50 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:25:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 12:25:51 Starting bucket ops to source bucket
2021/12/15 12:25:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:26:29 Waiting for dst bucket item count to get to: 5000 curr count: 2281
2021/12/15 12:26:34 Waiting for dst bucket item count to get to: 5000 curr count: 2281
2021/12/15 12:26:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:26:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:26:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:26:39 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 12:26:44 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 12:26:49 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 12:26:54 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/15 12:26:54 App: TestTimerInPastBucketOp got un-deployed
2021/12/15 12:26:54 No eventing-con process running
2021/12/15 12:26:54 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/15 12:26:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:27:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 12:27:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:27:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:27:18 Waiting for dst bucket item count to get to: 0 curr count: 2809
2021/12/15 12:27:23 Waiting for dst bucket item count to get to: 0 curr count: 309
2021/12/15 12:27:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (116.58s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/15 12:27:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:27:33 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:27:33 No eventing-con process running
2021/12/15 12:27:33 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/15 12:27:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:27:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:27:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:27:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:27:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 12:27:48 Starting bucket ops to source bucket
2021/12/15 12:27:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:28:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:28:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:28:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:28:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:28:21 Waiting for dst bucket item count to get to: 5000 curr count: 1695
2021/12/15 12:28:26 Waiting for dst bucket item count to get to: 5000 curr count: 1695
2021/12/15 12:28:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:28:31 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 12:28:31 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 12:28:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:28:31 Undeploying app: bucket_op_with_timer
2021/12/15 12:28:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:28:36 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:28:41 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:28:46 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:28:51 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:28:51 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:28:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:28:51 No eventing-con process running
2021/12/15 12:28:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:29:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:29:35 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:29:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 12:29:36 Starting bucket ops to source bucket
2021/12/15 12:29:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:29:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:29:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:29:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:30:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:30:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:30:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:30:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:30:19 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 12:30:19 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 12:30:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:30:19 Undeploying app: bucket_op_with_timer
2021/12/15 12:30:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:30:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:30:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:30:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:30:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:30:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:30:39 No eventing-con process running
2021/12/15 12:30:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:30:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:31:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:31:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 12:31:24 Starting bucket ops to source bucket
2021/12/15 12:31:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:31:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:31:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:31:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:31:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:31:57 Waiting for dst bucket item count to get to: 5000 curr count: 1227
2021/12/15 12:32:02 Waiting for dst bucket item count to get to: 5000 curr count: 2509
2021/12/15 12:32:07 Waiting for dst bucket item count to get to: 5000 curr count: 3604
2021/12/15 12:32:12 Waiting for dst bucket item count to get to: 5000 curr count: 4822
2021/12/15 12:32:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:32:17 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 12:32:17 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 12:32:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:32:17 Undeploying app: bucket_op_with_timer
2021/12/15 12:32:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:32:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:32:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:32:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:32:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:32:37 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:32:37 No eventing-con process running
2021/12/15 12:32:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:33:21 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:33:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 12:33:23 Starting bucket ops to source bucket
2021/12/15 12:33:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:33:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:33:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:33:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:33:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:33:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:34:01 Waiting for dst bucket item count to get to: 5000 curr count: 470
2021/12/15 12:34:06 Waiting for dst bucket item count to get to: 5000 curr count: 2970
2021/12/15 12:34:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:34:11 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 12:34:11 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 12:34:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:34:11 Undeploying app: bucket_op_with_timer
2021/12/15 12:34:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:34:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:34:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:34:26 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:34:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:34:31 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:34:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:34:31 No eventing-con process running
2021/12/15 12:34:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:34:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:35:15 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:35:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 12:35:16 Starting bucket ops to source bucket
2021/12/15 12:35:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:35:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:35:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:35:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:35:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:35:49 Waiting for dst bucket item count to get to: 5000 curr count: 1044
2021/12/15 12:35:54 Waiting for dst bucket item count to get to: 5000 curr count: 1044
2021/12/15 12:35:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:35:59 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/15 12:35:59 App: TestDeployUndeployLoopTimer got deployed
2021/12/15 12:35:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:35:59 Undeploying app: bucket_op_with_timer
2021/12/15 12:35:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:36:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:36:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:36:14 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:36:19 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/15 12:36:19 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/15 12:36:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:36:19 No eventing-con process running
2021/12/15 12:36:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:36:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:37:03 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.61s)
=== RUN   TestMultipleHandlers
2021/12/15 12:37:13 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 12:37:13 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 12:37:13 No eventing-con process running
2021/12/15 12:37:13 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/15 12:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:37:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:37:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:37:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:37:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 12:37:32 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 12:37:32 No eventing-con process running
2021/12/15 12:37:32 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/15 12:37:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:37:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:37:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:37:46 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:37:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 12:37:47 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:37:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 12:37:53 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 12:37:58 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 12:38:03 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 12:38:08 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/15 12:38:08 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/15 12:38:13 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 12:38:18 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 12:38:23 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/15 12:38:23 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/15 12:38:23 Starting bucket ops to source bucket
2021/12/15 12:38:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:38:38 Waiting for dst bucket item count to get to: 5000 curr count: 4610
2021/12/15 12:38:43 Waiting for dst bucket item count to get to: 5000 curr count: 4610
2021/12/15 12:38:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:38:43 Starting bucket ops to source bucket
2021/12/15 12:38:58 Waiting for dst bucket item count to get to: 10000 curr count: 5653
2021/12/15 12:39:03 Waiting for dst bucket item count to get to: 10000 curr count: 5653
2021/12/15 12:39:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 12:39:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:39:03 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:39:03 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:39:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:39:08 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 12:39:13 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 12:39:18 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 12:39:23 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/15 12:39:23 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/15 12:39:23 No eventing-con process running
2021/12/15 12:39:24 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/15 12:39:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:39:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:39:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:39:43 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 12:39:48 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/15 12:39:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:39:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:39:53 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 12:39:58 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 12:40:03 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 12:40:08 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/15 12:40:08 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/15 12:40:08 No eventing-con process running
2021/12/15 12:40:08 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/15 12:40:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:40:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:40:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (197.96s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/15 12:40:31 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 12:40:31 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 12:40:31 No eventing-con process running
2021/12/15 12:40:31 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/15 12:40:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:40:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:40:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:40:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:40:45 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:40:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:40:46 Starting bucket ops to source bucket
2021/12/15 12:40:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:40:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:41:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:41:09 Waiting for dst bucket item count to get to: 5000 curr count: 1924
2021/12/15 12:41:14 Waiting for dst bucket item count to get to: 5000 curr count: 1924
2021/12/15 12:41:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:41:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:41:19 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 12:41:19 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 12:41:19 Pausing app: bucket_op_on_update
2021/12/15 12:41:19 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:41:24 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 12:41:24 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 12:41:24 Resuming app: bucket_op_on_update
2021/12/15 12:41:25 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:41:25 Starting bucket ops to source bucket
2021/12/15 12:41:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:41:38 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:41:43 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:41:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 12:41:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:41:48 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 12:41:48 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 12:41:48 Pausing app: bucket_op_on_update
2021/12/15 12:41:48 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:41:53 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 12:41:53 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 12:41:53 Resuming app: bucket_op_on_update
2021/12/15 12:41:55 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:41:55 Starting bucket ops to source bucket
2021/12/15 12:42:02 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:42:07 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:42:12 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:42:13 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 12:42:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:42:18 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 12:42:18 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 12:42:18 Pausing app: bucket_op_on_update
2021/12/15 12:42:18 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:42:23 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 12:42:23 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 12:42:23 Resuming app: bucket_op_on_update
2021/12/15 12:42:24 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:42:24 Starting bucket ops to source bucket
2021/12/15 12:42:32 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:42:37 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:42:42 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:42:47 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:42:47 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 12:42:47 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:42:52 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 12:42:52 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 12:42:52 Pausing app: bucket_op_on_update
2021/12/15 12:42:52 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:42:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 12:42:57 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 12:42:57 Resuming app: bucket_op_on_update
2021/12/15 12:42:58 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:42:58 Starting bucket ops to source bucket
2021/12/15 12:43:06 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:43:11 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:43:16 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:43:16 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 12:43:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:43:21 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/15 12:43:21 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/15 12:43:21 Pausing app: bucket_op_on_update
2021/12/15 12:43:21 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:43:26 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/15 12:43:26 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/15 12:43:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:43:31 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 12:43:36 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 12:43:41 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 12:43:46 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/15 12:43:46 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/15 12:43:46 No eventing-con process running
2021/12/15 12:43:46 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/15 12:43:48 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:43:58 Waiting for dst bucket item count to get to: 0 curr count: 19513
2021/12/15 12:44:03 Waiting for dst bucket item count to get to: 0 curr count: 7013
2021/12/15 12:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:44:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:44:15 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 12:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.86s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/15 12:44:25 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 12:44:25 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 12:44:25 No eventing-con process running
2021/12/15 12:44:25 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/15 12:44:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:44:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:44:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:44:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:44:39 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:44:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 12:44:40 Starting bucket ops to source bucket
2021/12/15 12:44:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:44:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:44:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:45:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:45:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:45:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:45:18 Waiting for dst bucket item count to get to: 5000 curr count: 1308
2021/12/15 12:45:23 Waiting for dst bucket item count to get to: 5000 curr count: 3808
2021/12/15 12:45:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:45:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:45:28 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 12:45:28 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 12:45:28 Pausing app: bucket_op_with_timer
2021/12/15 12:45:28 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:45:33 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 12:45:33 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 12:45:33 Resuming app: bucket_op_with_timer
2021/12/15 12:45:34 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:45:34 Starting bucket ops to source bucket
2021/12/15 12:45:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:45:47 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:45:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:45:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:46:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:46:07 Waiting for dst bucket item count to get to: 10000 curr count: 9987
2021/12/15 12:46:12 Waiting for dst bucket item count to get to: 10000 curr count: 9987
2021/12/15 12:46:12 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 12:46:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:46:17 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 12:46:17 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 12:46:17 Pausing app: bucket_op_with_timer
2021/12/15 12:46:17 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:46:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 12:46:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 12:46:22 Resuming app: bucket_op_with_timer
2021/12/15 12:46:24 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:46:24 Starting bucket ops to source bucket
2021/12/15 12:46:32 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:46:37 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:46:42 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:46:47 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:46:52 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/15 12:46:52 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/15 12:46:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:46:57 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 12:46:57 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 12:46:57 Pausing app: bucket_op_with_timer
2021/12/15 12:46:57 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:47:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 12:47:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 12:47:02 Resuming app: bucket_op_with_timer
2021/12/15 12:47:03 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:47:03 Starting bucket ops to source bucket
2021/12/15 12:47:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:16 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:21 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:26 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:31 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:36 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/15 12:47:36 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 12:47:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:47:41 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 12:47:41 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 12:47:41 Pausing app: bucket_op_with_timer
2021/12/15 12:47:41 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:47:46 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 12:47:46 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 12:47:46 Resuming app: bucket_op_with_timer
2021/12/15 12:47:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:47:47 Starting bucket ops to source bucket
2021/12/15 12:47:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:48:00 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:48:05 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:48:10 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:48:15 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/15 12:48:20 Waiting for dst bucket item count to get to: 25000 curr count: 21137
2021/12/15 12:48:25 Waiting for dst bucket item count to get to: 25000 curr count: 21137
2021/12/15 12:48:25 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 12:48:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:48:30 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/15 12:48:30 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/15 12:48:30 Pausing app: bucket_op_with_timer
2021/12/15 12:48:30 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:48:35 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/15 12:48:35 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/15 12:48:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:48:40 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 12:48:45 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 12:48:50 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 12:48:55 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/15 12:48:55 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/15 12:48:55 No eventing-con process running
2021/12/15 12:48:55 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/15 12:48:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:49:07 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 12:49:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:49:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:49:19 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 12:49:24 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/15 12:49:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (308.90s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/15 12:49:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 12:49:34 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 12:49:34 No eventing-con process running
2021/12/15 12:49:34 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/15 12:49:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:49:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:49:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:49:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:49:48 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:49:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/15 12:49:54 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 12:49:59 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 12:50:04 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 12:50:09 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/15 12:50:09 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/15 12:50:09 Starting bucket ops to source bucket
2021/12/15 12:50:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:50:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:50:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:50:38 Waiting for dst bucket item count to get to: 5000 curr count: 978
2021/12/15 12:50:43 Waiting for dst bucket item count to get to: 5000 curr count: 2271
2021/12/15 12:50:48 Waiting for dst bucket item count to get to: 5000 curr count: 3793
2021/12/15 12:50:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:50:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:50:48 Pausing app: bucket_op_with_timer
2021/12/15 12:50:48 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:50:53 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/15 12:50:53 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/15 12:50:53 Resuming app: bucket_op_with_timer
2021/12/15 12:50:54 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:50:54 Starting bucket ops to source bucket
2021/12/15 12:51:06 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:51:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:51:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:51:21 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:51:26 Waiting for dst bucket item count to get to: 10000 curr count: 5927
2021/12/15 12:51:31 Waiting for dst bucket item count to get to: 10000 curr count: 8195
2021/12/15 12:51:36 Waiting for dst bucket item count to get to: 10000 curr count: 9768
2021/12/15 12:51:36 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 12:51:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:51:41 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 12:51:46 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 12:51:51 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 12:51:56 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/15 12:51:56 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/15 12:51:56 No eventing-con process running
2021/12/15 12:51:57 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/15 12:51:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:52:08 Waiting for dst bucket item count to get to: 0 curr count: 7581
2021/12/15 12:52:13 Waiting for dst bucket item count to get to: 0 curr count: 2581
2021/12/15 12:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:52:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:52:25 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 12:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (181.00s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/15 12:52:35 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 12:52:35 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 12:52:35 No eventing-con process running
2021/12/15 12:52:35 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/15 12:52:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:52:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:52:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:52:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:52:49 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:52:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 12:52:55 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 12:53:00 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 12:53:05 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 12:53:10 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/15 12:53:10 App: TestPauseResumeWithEventingReb got deployed
2021/12/15 12:53:10 Starting bucket ops to source bucket
2021/12/15 12:53:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:53:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:53:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:53:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:53:39 Waiting for dst bucket item count to get to: 5000 curr count: 1540
2021/12/15 12:53:44 Waiting for dst bucket item count to get to: 5000 curr count: 3080
2021/12/15 12:53:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:53:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:53:44 Pausing app: bucket_op_with_timer
2021/12/15 12:53:44 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:53:49 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/15 12:53:49 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/15 12:53:49 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 12:53:50 Removing node(s): [] from the cluster
2021/12/15 12:53:55 Rebalance progress: 29.16666666666666
2021/12/15 12:54:00 Rebalance progress: 29.16666666666666
2021/12/15 12:54:05 Rebalance progress: 100
2021/12/15 12:54:05 Resuming app: bucket_op_with_timer
2021/12/15 12:54:06 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:54:06 Starting bucket ops to source bucket
2021/12/15 12:54:18 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:54:23 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:54:28 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:54:33 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/15 12:54:38 Waiting for dst bucket item count to get to: 10000 curr count: 6005
2021/12/15 12:54:43 Waiting for dst bucket item count to get to: 10000 curr count: 6005
2021/12/15 12:54:43 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 12:54:43 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 12:54:48 Rebalance progress: 29.16666666666666
2021/12/15 12:54:53 Rebalance progress: 29.16666666666666
2021/12/15 12:54:58 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 12:55:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:55:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:55:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:55:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:56:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:56:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:56:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:56:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:57:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:57:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:57:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 12:57:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:65} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 216f6a46d929b7a3398e76a9d5fa4ec8
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: c627d24e79eddfbcb58cfabf855d16ae
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 0d293a0ed02efd028dc5cf9f94438737
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]


2021/12/15 12:58:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:58:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 12:58:09 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 12:58:14 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 12:58:19 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 12:58:24 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/15 12:58:24 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/15 12:58:24 eventingSetup.go:46: No eventing-con process running
2021/12/15 12:58:24 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/15 12:58:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:58:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 12:58:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:58:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:58:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 12:58:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (382.25s)
=== RUN   TestBucketDeleteAfterPause
2021/12/15 12:58:57 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 12:58:57 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 12:58:57 eventingSetup.go:46: No eventing-con process running
2021/12/15 12:58:57 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/15 12:58:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:59:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:59:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 12:59:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 12:59:11 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/15 12:59:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/15 12:59:17 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 12:59:22 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 12:59:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 12:59:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/15 12:59:32 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/15 12:59:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 12:59:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 12:59:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 12:59:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 12:59:42 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/15 12:59:42 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 12:59:47 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/15 12:59:47 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/15 12:59:47 handler_test.go:716: Deleting source bucket: default
2021/12/15 12:59:53 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/15 12:59:53 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/15 12:59:58 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/15 12:59:58 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/15 12:59:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 12:59:58 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/15 13:00:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:00:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:00:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:00:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.08s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/15 13:00:26 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 13:00:26 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 13:00:26 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:00:26 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/15 13:00:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:00:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:00:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:00:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:00:40 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:00:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 13:00:46 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 13:00:51 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 13:00:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 13:01:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/15 13:01:01 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/15 13:01:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:01:36 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/15 13:01:36 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:01:41 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/15 13:01:41 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/15 13:04:41 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/15 13:04:41 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/15 13:04:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/15 13:04:47 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 13:04:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:04:47 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/15 13:04:52 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 13:04:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:04:52 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2892 expected 5000
2021/12/15 13:04:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 13:04:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:04:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/15 13:05:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/15 13:05:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:05:02 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/15 13:05:02 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/15 13:05:02 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:05:12 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 13:05:17 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 13:05:22 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/15 13:05:22 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/15 13:05:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:05:22 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/15 13:05:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:05:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:05:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:05:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:05:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.87s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/15 13:05:55 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 13:05:55 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 13:05:55 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:05:55 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/15 13:05:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:06:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:06:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:06:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:06:08 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:06:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 13:06:15 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:06:20 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:06:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:06:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:06:30 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 13:06:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:06:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:06:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:06:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/15 13:06:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:06:49 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:06:49 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 13:06:49 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:07:19 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/15 13:07:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 13:07:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:07:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:07:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:07:40 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/15 13:07:40 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/15 13:07:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:07:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:07:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 454
2021/12/15 13:08:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 454
2021/12/15 13:08:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:08:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:08:03 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/15 13:08:03 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:08:13 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 13:08:18 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 13:08:23 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/15 13:08:23 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/15 13:08:23 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:08:23 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/15 13:08:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:08:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:08:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (170.85s)
=== RUN   TestCPPWorkerCleanup
2021/12/15 13:08:46 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 13:08:46 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 13:08:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:08:46 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/15 13:08:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:08:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:08:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:08:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:08:59 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:09:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/15 13:09:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:09:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:09:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:09:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:09:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:09:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4610
2021/12/15 13:09:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4610
2021/12/15 13:09:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:09:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:09:33 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:09:38 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 13:09:43 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 13:09:48 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 13:09:53 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/15 13:09:53 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/15 13:09:53 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:09:53 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/15 13:09:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:10:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:10:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:10:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:10:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:10:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4841
2021/12/15 13:10:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4841
2021/12/15 13:10:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.84s)
=== RUN   TestWithUserXattrs
2021/12/15 13:11:06 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 13:11:06 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 13:11:06 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:11:06 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/15 13:11:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:11:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:11:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:11:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:11:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:11:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:11:23 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:11:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/15 13:11:29 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 13:11:34 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 13:11:39 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 13:11:44 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/15 13:11:44 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/15 13:11:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:11:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:11:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4095
2021/12/15 13:12:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4095
2021/12/15 13:12:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:12:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:12:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:12:09 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 13:12:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 13:12:19 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 13:12:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/15 13:12:24 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/15 13:12:24 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:12:24 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/15 13:12:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:12:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:12:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:12:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:12:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:12:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.00s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/15 13:12:57 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:12:57 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 13:12:57 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:12:57 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/15 13:12:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:13:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:13:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:13:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:13:10 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:13:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/15 13:13:16 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 13:13:21 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 13:13:26 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 13:13:31 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/15 13:13:31 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/15 13:13:31 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:13:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:13:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:13:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:13:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:13:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:13:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3702
2021/12/15 13:14:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1202
2021/12/15 13:14:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:14:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:14:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:14:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:14:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:14:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14506
2021/12/15 13:14:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 14506
2021/12/15 13:14:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/15 13:14:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:14:47 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:14:52 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:14:57 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:15:02 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:15:07 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:15:07 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 13:15:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:15:12 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/15 13:15:12 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/15 13:15:12 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:15:12 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/15 13:15:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:15:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/15 13:15:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/15 13:15:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:15:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:15:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 15402
2021/12/15 13:15:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 15402
2021/12/15 13:15:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.95s)
=== RUN   TestMetaBucketDelete
2021/12/15 13:15:55 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 13:15:55 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 13:15:55 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:15:55 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/15 13:15:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:16:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:16:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:16:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:16:08 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:16:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 13:16:14 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 13:16:19 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 13:16:24 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 13:16:29 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/15 13:16:29 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/15 13:16:29 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/15 13:16:30 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/15 13:16:35 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 13:16:35 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 13:16:50 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/15 13:16:50 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/15 13:16:50 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:16:51 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/15 13:16:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:16:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:16:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:17:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.66s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/15 13:17:14 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:17:14 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:17:14 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:17:14 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 13:17:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:17:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:17:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:17:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:17:27 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:17:28 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 13:17:47 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/15 13:17:47 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/15 13:18:02 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:18:02 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:18:17 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:18:17 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:18:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:18:17 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 13:18:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:18:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:18:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:18:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.82s)
=== RUN   TestSourceBucketDelete
2021/12/15 13:18:40 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 13:18:40 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 13:18:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:18:40 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/15 13:18:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:18:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:18:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:18:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:18:54 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:18:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/15 13:19:00 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 13:19:05 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 13:19:10 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 13:19:15 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/15 13:19:15 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/15 13:19:15 handler_test.go:945: Deleting source bucket: default
2021/12/15 13:19:16 handler_test.go:947: Deleted source bucket: default
2021/12/15 13:19:21 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 13:19:22 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 13:19:37 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/15 13:19:37 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/15 13:19:37 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:19:37 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/15 13:19:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:19:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:19:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:19:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.42s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/15 13:20:00 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:20:00 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:20:00 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:20:00 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 13:20:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:20:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:20:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:20:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:20:13 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:20:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 13:20:33 handler_test.go:966: Deleting source bucket: default
2021/12/15 13:20:34 handler_test.go:968: Deleted source bucket: default
2021/12/15 13:20:49 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:20:49 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:21:04 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:21:04 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:21:04 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:21:04 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 13:21:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:21:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:21:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:21:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.54s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/15 13:21:27 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:21:27 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:21:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:21:27 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/15 13:21:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:21:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:21:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:21:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:21:40 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:21:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 13:22:00 handler_test.go:988: Deleting source bucket: default
2021/12/15 13:22:02 handler_test.go:990: Deleted source bucket: default
2021/12/15 13:22:02 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/15 13:22:03 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/15 13:22:18 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:22:18 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:22:33 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/15 13:22:33 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/15 13:22:33 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:22:33 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/15 13:22:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:22:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:22:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:22:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.30s)
=== RUN   TestUndeployDuringBootstrap
2021/12/15 13:22:56 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 13:22:56 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 13:22:56 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:22:56 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/15 13:22:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:23:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:23:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:23:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:23:09 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:23:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 13:23:10 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:23:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:23:22 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:23:22 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 13:23:22 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/15 13:23:22 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 13:23:27 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/15 13:23:32 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/15 13:23:37 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 13:23:42 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 13:23:47 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 13:23:47 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 13:23:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:24:12 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/15 13:24:12 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/15 13:24:12 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:24:12 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/15 13:24:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:24:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:24:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:24:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:24:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:24:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2828
2021/12/15 13:24:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2828
2021/12/15 13:24:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.77s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/15 13:24:50 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:24:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/15 13:24:57 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 13:25:02 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 13:25:07 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 13:25:12 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/15 13:25:12 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/15 13:25:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/15 13:25:12 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:25:12 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/15 13:25:17 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/15 13:25:17 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/15 13:25:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:25:17 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/15 13:25:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:25:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:25:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:25:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.57s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/15 13:25:40 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:25:40 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 13:25:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:25:40 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/15 13:25:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:25:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:25:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:25:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:25:53 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:25:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/15 13:26:00 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 13:26:05 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 13:26:10 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 13:26:15 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/15 13:26:15 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/15 13:26:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:26:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:26:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:26:55 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:27:00 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:27:05 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:27:05 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 13:27:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:28:50 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/15 13:28:50 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/15 13:28:50 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:28:50 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/15 13:28:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:29:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:29:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:29:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:29:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:29:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3377
2021/12/15 13:29:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 877
2021/12/15 13:29:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (232.99s)
=== RUN   TestUndeployWithKVFailover
2021/12/15 13:29:33 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 13:29:33 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/15 13:29:33 eventingSetup.go:46: No eventing-con process running
2021/12/15 13:29:33 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/15 13:29:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:29:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:29:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:29:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:29:46 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:29:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/15 13:29:47 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 13:29:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:29:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:30:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:30:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 13:30:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 13:30:30 Rebalance progress: 29.16666666666666
2021/12/15 13:30:35 Rebalance progress: 29.16666666666666
2021/12/15 13:30:40 Rebalance progress: 100
2021/12/15 13:30:40 Failing over: [127.0.0.1:19003]
2021/12/15 13:30:50 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/15 13:31:50 Removing node(s): [] from the cluster
2021/12/15 13:31:55 Rebalance progress: 29.16666666666666
2021/12/15 13:32:00 Rebalance progress: 29.16666666666666
2021/12/15 13:32:05 Rebalance progress: 100
2021/12/15 13:32:10 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 13:32:10 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 13:32:10 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:32:15 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/15 13:32:15 App: TestUndeployWithKVFailover got un-deployed
2021/12/15 13:32:15 No eventing-con process running
2021/12/15 13:32:15 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/15 13:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:32:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:32:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:32:38 Waiting for dst bucket item count to get to: 0 curr count: 3518
2021/12/15 13:32:43 Waiting for dst bucket item count to get to: 0 curr count: 1018
2021/12/15 13:32:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.10s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/15 13:32:53 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 13:32:53 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 13:32:53 No eventing-con process running
2021/12/15 13:32:53 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/15 13:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:33:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:33:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:33:06 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:33:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/15 13:33:12 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 13:33:17 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 13:33:22 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 13:33:27 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/15 13:33:27 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/15 13:33:27 Starting bucket ops to source bucket
2021/12/15 13:33:48 Waiting for dst bucket item count to get to: 20000 curr count: 13704
2021/12/15 13:33:53 Waiting for dst bucket item count to get to: 20000 curr count: 13704
2021/12/15 13:33:53 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/15 13:33:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:33:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:34:06 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/15 13:34:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:34:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:34:18 Waiting for dst bucket item count to get to: 0 curr count: 10702
2021/12/15 13:34:23 Waiting for dst bucket item count to get to: 0 curr count: 702
2021/12/15 13:34:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:34:23 Starting bucket ops to source bucket
2021/12/15 13:34:36 Waiting for dst bucket item count to get to: 10000 curr count: 2831
2021/12/15 13:34:41 Waiting for dst bucket item count to get to: 10000 curr count: 7012
2021/12/15 13:34:46 Waiting for dst bucket item count to get to: 10000 curr count: 9181
2021/12/15 13:34:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 13:34:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:34:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:34:51 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 13:34:56 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 13:35:01 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 13:35:06 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/15 13:35:06 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/15 13:35:06 No eventing-con process running
2021/12/15 13:35:06 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/15 13:35:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:35:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 13:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:35:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:35:29 Waiting for dst bucket item count to get to: 0 curr count: 9838
2021/12/15 13:35:34 Waiting for dst bucket item count to get to: 0 curr count: 4838
2021/12/15 13:35:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (170.79s)
=== RUN   TestTimerOverwrite
2021/12/15 13:35:44 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 13:35:44 App: TestTimerOverwrite got un-deployed
2021/12/15 13:35:44 No eventing-con process running
2021/12/15 13:35:44 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/15 13:35:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:35:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:35:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:35:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:35:57 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:35:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/15 13:36:03 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 13:36:08 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 13:36:13 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 13:36:18 Waiting for app: TestTimerOverwrite to get deployed
2021/12/15 13:36:18 App: TestTimerOverwrite got deployed
2021/12/15 13:36:28 Starting bucket ops to source bucket
2021/12/15 13:36:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:36:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:36:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:36:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:36:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 13:37:38 Waiting for dst bucket item count to get to: 5000 curr count: 2516
2021/12/15 13:37:38 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:37:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:37:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:37:53 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 13:37:58 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 13:38:03 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 13:38:08 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/15 13:38:08 App: TestTimerOverwrite got un-deployed
2021/12/15 13:38:08 No eventing-con process running
2021/12/15 13:38:08 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/15 13:38:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:38:19 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:38:24 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:38:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:38:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:38:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (182.29s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/15 13:38:41 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 13:38:41 Removing node(s): [] from the cluster
2021/12/15 13:38:46 Rebalance progress: 100
2021/12/15 13:38:46 Failing over: [127.0.0.1:19003]
2021/12/15 13:38:51 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 13:38:51 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 13:38:51 No eventing-con process running
2021/12/15 13:38:51 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 13:38:53 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:39:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:39:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:39:05 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:39:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/15 13:39:06 Starting bucket ops to source bucket
2021/12/15 13:39:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:39:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:39:16 Context: Post to set retry counter, response code: 200 dump: 
2021/12/15 13:39:16 set retry counter, response &{[] }
2021/12/15 13:40:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 13:40:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 13:40:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:40:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/15 13:40:26 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/15 13:40:26 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/15 13:40:26 No eventing-con process running
2021/12/15 13:40:26 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/15 13:40:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:40:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:40:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:40:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:40:49 Waiting for dst bucket item count to get to: 0 curr count: 4139
2021/12/15 13:40:54 Waiting for dst bucket item count to get to: 0 curr count: 1639
2021/12/15 13:40:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:40:54 Removing node(s): [] from the cluster
2021/12/15 13:40:59 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.80s)
=== RUN   TestOnUpdateSrcMutation
2021/12/15 13:41:09 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 13:41:09 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 13:41:09 No eventing-con process running
2021/12/15 13:41:09 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/15 13:41:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:41:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:41:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:41:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:41:22 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:41:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 13:41:23 Starting bucket ops to source bucket
2021/12/15 13:41:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 13:41:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:41:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:41:50 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 13:41:55 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 13:42:00 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 13:42:05 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/15 13:42:05 App: TestOnUpdateSrcMutation got un-deployed
2021/12/15 13:42:05 No eventing-con process running
2021/12/15 13:42:05 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/15 13:42:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:42:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 13:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:42:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:42:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.52s)
=== RUN   TestOnDeleteSrcMutation
2021/12/15 13:42:33 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 13:42:33 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 13:42:34 No eventing-con process running
2021/12/15 13:42:34 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/15 13:42:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:42:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:42:42 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:42:47 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:42:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/15 13:42:48 Starting bucket ops to source bucket
2021/12/15 13:43:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:43:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:43:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:43:22 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 13:43:27 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 13:43:32 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 13:43:37 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/15 13:43:37 App: TestOnDeleteSrcMutation got un-deployed
2021/12/15 13:43:37 No eventing-con process running
2021/12/15 13:43:37 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/15 13:43:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:43:48 Waiting for dst bucket item count to get to: 0 curr count: 3538
2021/12/15 13:43:53 Waiting for dst bucket item count to get to: 0 curr count: 1038
2021/12/15 13:43:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:43:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:44:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.28s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/15 13:44:10 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 13:44:10 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 13:44:10 No eventing-con process running
2021/12/15 13:44:10 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/15 13:44:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:44:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:44:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:44:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:44:23 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:44:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 13:44:24 Starting bucket ops to source bucket
2021/12/15 13:45:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 13:45:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:45:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:45:11 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 13:45:16 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 13:45:21 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 13:45:26 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/15 13:45:26 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/15 13:45:26 No eventing-con process running
2021/12/15 13:45:27 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/15 13:45:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:45:38 Waiting for dst bucket item count to get to: 0 curr count: 5571
2021/12/15 13:45:43 Waiting for dst bucket item count to get to: 0 curr count: 571
2021/12/15 13:45:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:45:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:45:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.77s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/15 13:46:00 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 13:46:00 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 13:46:00 No eventing-con process running
2021/12/15 13:46:00 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/15 13:46:01 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:46:08 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:46:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:46:13 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:46:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/15 13:46:14 Starting bucket ops to source bucket
2021/12/15 13:46:58 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 13:46:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 13:46:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:47:03 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 13:47:08 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 13:47:13 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 13:47:18 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/15 13:47:18 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/15 13:47:18 No eventing-con process running
2021/12/15 13:47:18 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/15 13:47:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:47:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 13:47:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 13:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:47:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:47:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (111.33s)
=== RUN   TestInterHandlerRecursion
2021/12/15 13:47:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:47:51 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 13:47:51 No eventing-con process running
2021/12/15 13:47:51 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 13:47:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:47:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:47:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:48:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:48:09 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:48:09 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 13:48:09 No eventing-con process running
2021/12/15 13:48:09 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 13:48:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:48:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:48:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:48:22 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:48:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 13:48:23 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:48:28 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:48:33 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:48:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:48:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:48:43 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 13:48:43 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:48:43 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/15 13:48:43 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/15 13:48:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:48:48 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:48:53 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:48:58 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:49:03 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:49:04 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 13:49:04 No eventing-con process running
2021/12/15 13:49:04 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 13:49:05 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:49:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:49:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:49:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:49:22 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:49:22 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 13:49:22 No eventing-con process running
2021/12/15 13:49:22 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 13:49:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:49:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:49:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:49:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.80s)
=== RUN   TestInterBucketRecursion
2021/12/15 13:49:40 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:49:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 13:49:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:49:41 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:49:41 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 13:49:46 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 13:49:51 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 13:49:56 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 13:50:01 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/15 13:50:01 App: TestInterBucketRecursion_function1 got deployed
2021/12/15 13:50:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:50:06 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:50:11 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:50:16 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:50:21 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:50:21 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/15 13:50:21 No eventing-con process running
2021/12/15 13:50:21 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 13:50:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:50:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:50:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:50:39 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/15 13:50:39 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/15 13:50:39 No eventing-con process running
2021/12/15 13:50:39 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/15 13:50:40 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:50:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:50:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.59s)
=== RUN   TestLargeHandler
2021/12/15 13:50:52 Changed force_compress value to false
2021/12/15 13:50:52 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:50:52 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/15 13:50:52 Changed force_compress value to true
2021/12/15 13:50:52 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:50:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/15 13:50:59 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 13:51:04 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 13:51:09 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 13:51:14 Waiting for app: TestLargeHandler_function to get deployed
2021/12/15 13:51:14 App: TestLargeHandler_function got deployed
2021/12/15 13:51:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:51:19 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 13:51:25 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 13:51:30 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 13:51:35 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/15 13:51:35 App: TestLargeHandler_function got un-deployed
2021/12/15 13:51:35 No eventing-con process running
2021/12/15 13:51:35 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/15 13:51:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:51:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:51:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:51:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.40s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/15 13:51:58 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:51:58 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 13:51:58 No eventing-con process running
2021/12/15 13:51:58 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/15 13:51:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:52:16 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:52:16 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 13:52:16 No eventing-con process running
2021/12/15 13:52:16 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/15 13:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:52:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:52:29 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:52:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/15 13:52:30 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:52:35 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:52:40 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:52:45 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:52:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/15 13:52:50 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/15 13:52:50 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:52:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/15 13:52:51 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:52:56 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 13:53:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 13:53:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 13:53:11 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/15 13:53:11 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/15 13:53:11 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/15 13:53:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:53:17 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:53:22 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:53:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:53:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/15 13:53:32 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/15 13:53:32 No eventing-con process running
2021/12/15 13:53:32 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/15 13:53:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:53:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:53:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:53:51 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:53:56 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:54:01 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:54:06 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/15 13:54:06 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/15 13:54:06 No eventing-con process running
2021/12/15 13:54:06 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/15 13:54:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:54:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:54:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.32s)
=== RUN   TestAllowInterBucketRecursion
2021/12/15 13:54:24 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:54:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 13:54:25 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:54:30 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 13:54:35 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 13:54:40 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 13:54:45 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/15 13:54:45 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/15 13:54:45 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:54:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 13:54:46 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 13:54:51 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 13:54:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 13:55:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 13:55:06 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/15 13:55:06 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/15 13:55:06 TestAllowInterBucketRecursion_function2 is deployed
2021/12/15 13:55:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:55:11 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:55:16 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:55:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:55:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/15 13:55:27 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/15 13:55:27 No eventing-con process running
2021/12/15 13:55:27 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/15 13:55:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:55:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:55:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:55:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:55:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:55:46 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 13:55:51 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 13:55:56 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 13:56:01 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/15 13:56:01 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/15 13:56:01 No eventing-con process running
2021/12/15 13:56:01 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/15 13:56:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:56:09 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.39s)
=== RUN   TestInterBucketRecursion2
2021/12/15 13:56:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:32 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:56:43 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:56:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 13:56:50 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:56:55 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:00 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:05 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:10 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:15 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:20 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/15 13:57:20 App: TestInterBucketRecursion2_function1 got deployed
2021/12/15 13:57:20 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:57:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 13:57:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 13:57:31 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 13:57:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 13:57:41 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/15 13:57:41 App: TestInterBucketRecursion2_function2 got deployed
2021/12/15 13:57:41 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:57:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 13:57:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 13:57:52 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 13:57:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 13:58:02 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/15 13:58:02 App: TestInterBucketRecursion2_function3 got deployed
2021/12/15 13:58:02 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/15 13:58:02 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/15 13:58:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:58:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 13:58:12 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 13:58:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 13:58:22 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/15 13:58:22 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/15 13:58:22 No eventing-con process running
2021/12/15 13:58:22 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/15 13:58:24 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:58:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:58:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:58:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:58:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:58:41 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 13:58:46 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 13:58:51 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 13:58:56 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/15 13:58:56 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/15 13:58:56 No eventing-con process running
2021/12/15 13:58:56 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/15 13:58:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:59:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:59:04 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:59:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 13:59:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 13:59:19 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 13:59:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 13:59:29 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/15 13:59:29 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/15 13:59:29 No eventing-con process running
2021/12/15 13:59:29 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/15 13:59:31 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:59:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:59:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:59:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:59:47 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/15 13:59:47 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/15 13:59:48 No eventing-con process running
2021/12/15 13:59:48 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/15 13:59:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 13:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 13:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:00:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (228.20s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/15 14:00:08 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 14:00:13 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/15 14:00:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:00:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:00:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:01:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:01:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:01:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:01:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:02:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:02:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:02:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:02:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:03:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:03:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:03:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:03:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:04:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:04:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:04:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:04:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:05:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:05:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:05:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/15 14:05:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:50} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/15 14:06:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:06:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 14:06:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 14:06:06 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 14:06:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:06:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 14:06:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 14:06:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/15 14:06:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:06:17 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 14:06:17 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 14:06:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 14:06:17 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/15 14:06:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:06:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:06:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:06:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:06:34 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:06:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 14:06:41 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 14:06:46 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 14:06:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 14:06:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/15 14:06:56 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 0d293a0ed02efd028dc5cf9f94438737
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
Producer node: 127.0.0.1:9302	Node UUID: 216f6a46d929b7a3398e76a9d5fa4ec8
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: c627d24e79eddfbcb58cfabf855d16ae
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]


2021/12/15 14:06:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:06:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 14:07:01 Rebalance progress: 29.16666666666666
2021/12/15 14:07:06 Rebalance progress: 29.16666666666666
2021/12/15 14:07:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 216f6a46d929b7a3398e76a9d5fa4ec8
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: c627d24e79eddfbcb58cfabf855d16ae
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 0d293a0ed02efd028dc5cf9f94438737
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]


2021/12/15 14:07:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:07:11 handler_test.go:1674: Deleting source bucket: default
2021/12/15 14:07:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 14:07:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 14:07:18 Rebalance progress: 29.16666666666666
2021/12/15 14:07:23 Rebalance progress: 29.16666666666666
2021/12/15 14:07:28 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 904
	vb list dump: [0-13, 15-16, 18-25, 27-28, 30-42, 44-46, 48-68, 70-71, 73-75, 77-86, 89, 91-108, 110-113, 115-138, 140-148, 151-156, 158, 160-164, 166, 168-176, 178-189, 191-192, 194-196, 198-210, 212-214, 216-217, 219-238, 240-259, 261-269, 271-288, 291-304, 307-312, 314-317, 320-321, 323-332, 334, 336-349, 351-354, 358-375, 377-381, 383-391, 393-396, 398-399, 401-417, 419-427, 429, 431-438, 440-449, 451-455, 458-465, 468, 470-477, 479-481, 483, 485-486, 488, 490-492, 494-500, 502-503, 505-509, 511-572, 574-580, 582-586, 589, 591-592, 594-601, 603-620, 622-645, 647-650, 652-653, 655-665, 668-685, 687-688, 690-692, 694-702, 704-705, 707-711, 713, 715-749, 751-759, 761-780, 782-786, 789-792, 794-800, 802-804, 806-812, 814-823, 825-828, 831-851, 853-859, 861-874, 877-882, 884-889, 891-892, 894-912, 914-915, 917-919, 922-934, 936-939, 941-948, 951-954, 956-982, 984-989, 991-992, 994-1023]
	stream status: stopped
	len: 120
	vb list dump: [14, 17, 26, 29, 43, 47, 69, 72, 76, 87-88, 90, 109, 114, 139, 149-150, 157, 159, 165, 167, 177, 190, 193, 197, 211, 215, 218, 239, 260, 270, 289-290, 305-306, 313, 318-319, 322, 333, 335, 350, 355-357, 376, 382, 392, 397, 400, 418, 428, 430, 439, 450, 456-457, 466-467, 469, 478, 482, 484, 487, 489, 493, 501, 504, 510, 573, 581, 587-588, 590, 593, 602, 621, 646, 651, 654, 666-667, 686, 689, 693, 703, 706, 712, 714, 750, 760, 781, 787-788, 793, 801, 805, 813, 824, 829-830, 852, 860, 875-876, 883, 890, 893, 913, 916, 920-921, 935, 940, 949-950, 955, 983, 990, 993]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: 0d293a0ed02efd028dc5cf9f94438737
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-349, 351-354, 358-375, 377-381, 383-391, 393-396, 398-399, 401-417, 419-427, 429, 431-438, 440-449, 451-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 100
	v: [458-465, 468, 470-477, 479-481, 483, 485-486, 488, 490-492, 494-500, 502-503, 505-509, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-572, 574-580, 582-586, 589, 591-592, 594-601, 603-620, 622-645, 647-650, 652-653, 655-665, 668-682]
Producer node: 127.0.0.1:9302	Node UUID: 216f6a46d929b7a3398e76a9d5fa4ec8
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-13, 15-16, 18-25, 27-28, 30-42, 44-46, 48-68, 70-71, 73-75, 77-86, 89, 91-108, 110-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [115-138, 140-148, 151-156, 158, 160-164, 166, 168-176, 178-189, 191-192, 194-196, 198-210, 212-214, 216-217, 219-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-238, 240-259, 261-269, 271-288, 291-304, 307-312, 314-317, 320-321, 323-332, 334, 336-341]
Producer node: 172.31.2.134:9300	Node UUID: c627d24e79eddfbcb58cfabf855d16ae
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-912, 914-915, 917-919, 922-934, 936-939, 941-948, 951-954, 956-982, 984-989, 991-992, 994-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-685, 687-688, 690-692, 694-702, 704-705, 707-711, 713, 715-749, 751-759, 761-780, 782-786, 789-792, 794-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-800, 802-804, 806-812, 814-823, 825-828, 831-851, 853-859, 861-874, 877-882, 884-889, 891-892, 894-910]


2021/12/15 14:07:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:07:33 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/15 14:07:34 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/15 14:07:39 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/15 14:07:39 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/15 14:07:39 eventingSetup.go:46: No eventing-con process running
2021/12/15 14:07:39 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/15 14:07:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:07:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:07:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:07:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:07:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 14:07:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 14:07:57 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (477.37s)
=== RUN   TestN1QLRecursion
2021/12/15 14:08:00 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 14:08:00 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 14:08:00 Removing node(s): [] from the cluster
2021/12/15 14:08:05 Rebalance progress: 100
2021/12/15 14:08:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:08:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:08:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:08:28 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:08:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 14:08:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 14:08:34 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:39 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/15 14:08:59 App: TestN1QLRecursion_function1 got deployed
2021/12/15 14:08:59 TestN1QLRecursion_function1 is deployed
2021/12/15 14:08:59 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:09:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 14:09:01 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 14:09:06 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 14:09:11 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 14:09:16 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 14:09:21 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/15 14:09:21 App: TestN1QLRecursion_function3 got deployed
2021/12/15 14:09:21 TestN1QLRecursion_function3 is deployed
2021/12/15 14:09:21 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:09:21 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/15 14:09:21 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/15 14:09:21 Success: TestN1QLRecursion_function2 is not deployed
2021/12/15 14:09:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:09:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 14:09:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 14:09:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 14:09:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/15 14:09:41 App: TestN1QLRecursion_function1 got un-deployed
2021/12/15 14:09:41 No eventing-con process running
2021/12/15 14:09:41 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/15 14:09:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:09:49 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:09:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:09:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:10:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 14:10:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 14:10:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 14:10:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/15 14:10:15 App: TestN1QLRecursion_function3 got un-deployed
2021/12/15 14:10:15 No eventing-con process running
2021/12/15 14:10:15 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/15 14:10:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:10:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:10:23 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:10:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:10:30 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 14:10:35 Rebalance progress: 100
--- PASS: TestN1QLRecursion (154.79s)
=== RUN   TestN1QLAllowRecursion
2021/12/15 14:10:35 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 14:10:35 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 14:10:35 Removing node(s): [] from the cluster
2021/12/15 14:10:40 Rebalance progress: 100
2021/12/15 14:10:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:10:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:10:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:11:03 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:11:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/15 14:11:04 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/15 14:11:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 14:11:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 14:11:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 14:11:24 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/15 14:11:24 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/15 14:11:24 TestN1QLAllowRecursion_function1 is deployed
2021/12/15 14:11:24 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:11:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/15 14:11:25 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/15 14:11:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 14:11:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 14:11:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 14:11:45 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/15 14:11:45 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/15 14:11:45 TestN1QLAllowRecursion_function3 is deployed
2021/12/15 14:11:45 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:11:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/15 14:11:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/15 14:11:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 14:11:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 14:12:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 14:12:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/15 14:12:07 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/15 14:12:07 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/15 14:12:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:12:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 14:12:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 14:12:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 14:12:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/15 14:12:27 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/15 14:12:27 No eventing-con process running
2021/12/15 14:12:27 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/15 14:12:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:12:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:12:36 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:12:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:12:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:12:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 14:12:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 14:12:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 14:13:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/15 14:13:01 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/15 14:13:01 No eventing-con process running
2021/12/15 14:13:01 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/15 14:13:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:13:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:13:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:13:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:13:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 14:13:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 14:13:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 14:13:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/15 14:13:35 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/15 14:13:35 No eventing-con process running
2021/12/15 14:13:35 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/15 14:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:13:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:13:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:13:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:13:50 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 14:13:55 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.30s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/15 14:13:55 Sleeping for some time
2021/12/15 14:14:05 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 14:14:05 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 14:14:05 No eventing-con process running
2021/12/15 14:14:05 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/15 14:14:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:14:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:14:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:14:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:14:19 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:14:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 14:14:20 Starting bucket ops to source bucket
2021/12/15 14:15:09 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 14:15:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 14:15:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 14:15:14 Starting bucket ops to source bucket
2021/12/15 14:15:15 Starting bucket ops to source bucket
2021/12/15 14:16:21 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/15 14:16:21 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/15 14:16:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:16:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:16:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 14:16:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 14:16:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 14:16:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/15 14:16:41 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/15 14:16:41 No eventing-con process running
2021/12/15 14:16:41 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/15 14:16:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:16:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:16:58 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/15 14:16:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:17:00 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:17:10 Waiting for dst bucket item count to get to: 0 curr count: 4136
2021/12/15 14:17:15 Waiting for dst bucket item count to get to: 0 curr count: 4136
2021/12/15 14:17:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.86s)
=== RUN   TestTimerOverWriteSameReference
2021/12/15 14:17:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 14:17:20 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 14:17:20 No eventing-con process running
2021/12/15 14:17:20 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/15 14:17:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:17:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:17:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:17:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:17:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:17:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/15 14:17:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 14:17:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 14:17:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 14:17:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/15 14:17:55 App: TestTimerOverWriteSameReference got deployed
2021/12/15 14:18:05 Starting bucket ops to source bucket
2021/12/15 14:19:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/15 14:19:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:19:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:19:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 14:19:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 14:19:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 14:19:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/15 14:19:36 App: TestTimerOverWriteSameReference got un-deployed
2021/12/15 14:19:36 No eventing-con process running
2021/12/15 14:19:36 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/15 14:19:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:19:48 Waiting for dst bucket item count to get to: 0 curr count: 614
2021/12/15 14:19:53 Waiting for dst bucket item count to get to: 0 curr count: 114
2021/12/15 14:19:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:19:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:20:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/15 14:20:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.81s)
=== RUN   TestCancelTimerBucketop
2021/12/15 14:20:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 14:20:10 App: TestCancelTimerBucketop got un-deployed
2021/12/15 14:20:10 No eventing-con process running
2021/12/15 14:20:10 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/15 14:20:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:20:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:20:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:20:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:20:23 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:20:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/15 14:20:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 14:20:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 14:20:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 14:20:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/15 14:20:45 App: TestCancelTimerBucketop got deployed
2021/12/15 14:20:45 Starting bucket ops to source bucket
2021/12/15 14:20:52 Starting bucket ops to source bucket
2021/12/15 14:21:54 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 14:21:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:21:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:21:59 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 14:22:04 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 14:22:09 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 14:22:14 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/15 14:22:14 App: TestCancelTimerBucketop got un-deployed
2021/12/15 14:22:14 No eventing-con process running
2021/12/15 14:22:14 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/15 14:22:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:22:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:22:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:22:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:22:38 Waiting for dst bucket item count to get to: 0 curr count: 628
2021/12/15 14:22:43 Waiting for dst bucket item count to get to: 0 curr count: 128
2021/12/15 14:22:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.17s)
=== RUN   TestJSExpiryDate
2021/12/15 14:22:48 Starting bucket ops to source bucket
2021/12/15 14:23:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 14:23:06 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/15 14:23:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/15 14:23:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:23:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:23:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:23:42 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 14:23:47 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 14:23:52 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 14:23:57 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/15 14:23:57 App: TestJSExpiryDate got un-deployed
2021/12/15 14:23:57 No eventing-con process running
2021/12/15 14:23:57 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/15 14:23:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:24:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (88.67s)
=== RUN   TestBinaryDoc
2021/12/15 14:24:17 Starting bucket ops to source bucket
2021/12/15 14:24:17 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:24:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/15 14:24:18 Starting bucket ops to source bucket
2021/12/15 14:24:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:24:18 Deleting source bucket docs now
2021/12/15 14:24:18 Starting bucket ops to source bucket
2021/12/15 14:24:55 Waiting for dst bucket item count to get to: 32 curr count: 20
2021/12/15 14:25:00 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/15 14:25:05 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/15 14:25:05 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/15 14:25:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:25:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:25:10 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 14:25:15 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 14:25:20 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 14:25:25 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/15 14:25:25 App: TestBinaryDoc got un-deployed
2021/12/15 14:25:25 No eventing-con process running
2021/12/15 14:25:25 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/15 14:25:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:25:37 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/15 14:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:25:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:25:49 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/15 14:25:54 Waiting for dst bucket item count to get to: 0 curr count: 14
2021/12/15 14:25:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (102.32s)
=== RUN   TestConstantBindings
2021/12/15 14:25:54 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:25:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/15 14:25:55 Starting bucket ops to source bucket
2021/12/15 14:26:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 14:26:06 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 14:26:11 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/15 14:26:16 Waiting for dst bucket item count to get to: 6000 curr count: 792
2021/12/15 14:26:21 Waiting for dst bucket item count to get to: 6000 curr count: 3792
2021/12/15 14:26:21 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/15 14:26:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:26:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:26:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 14:26:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 14:26:36 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 14:26:41 Waiting for app: TestConstantBindings to get un-deployed
2021/12/15 14:26:41 App: TestConstantBindings got un-deployed
2021/12/15 14:26:41 No eventing-con process running
2021/12/15 14:26:41 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/15 14:26:43 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:26:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:26:58 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 14:26:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:26:59 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:27:09 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/15 14:27:14 Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/15 14:27:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.64s)
=== RUN   TestCodeInsight
2021/12/15 14:27:19 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 14:27:20 App: TestCodeInsight got un-deployed
2021/12/15 14:27:20 No eventing-con process running
2021/12/15 14:27:20 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/15 14:27:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:27:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:27:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:27:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:27:33 Starting bucket ops to source bucket
2021/12/15 14:27:34 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:27:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/15 14:27:40 Waiting for app: TestCodeInsight to get deployed
2021/12/15 14:27:45 Waiting for app: TestCodeInsight to get deployed
2021/12/15 14:27:50 Waiting for app: TestCodeInsight to get deployed
2021/12/15 14:27:55 Waiting for app: TestCodeInsight to get deployed
2021/12/15 14:27:55 App: TestCodeInsight got deployed
2021/12/15 14:28:00 Waiting for dst bucket item count to get to: 1000 curr count: 658
2021/12/15 14:28:00 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 14:28:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:28:05 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 14:28:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 14:28:15 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 14:28:20 Waiting for app: TestCodeInsight to get un-deployed
2021/12/15 14:28:20 App: TestCodeInsight got un-deployed
2021/12/15 14:28:20 No eventing-con process running
2021/12/15 14:28:20 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/15 14:28:22 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:28:32 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:28:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:28:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:28:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:28:49 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:28:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 14:28:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.47s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.06s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	13281.997s

Functional tests took 3 hours, 41 minutes and 27 seconds.

N1ql tests

2021/12/15 14:29:04 Started n1ql
2021/12/15 14:29:14 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/15 14:29:14 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/15 14:29:14 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 14:29:14 Removing node(s): [] from the cluster
2021/12/15 14:29:20 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/15 14:29:25 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/15 14:29:25 App: TestRecursiveMutationN1QL got un-deployed
2021/12/15 14:29:25 No eventing-con process running
2021/12/15 14:29:25 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/15 14:29:26 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:29:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:29:38 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:29:38 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.61s)
=== RUN   TestFlexReset1
2021/12/15 14:29:43 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 14:29:43 App: TestFlexReset1 got un-deployed
2021/12/15 14:29:43 No eventing-con process running
2021/12/15 14:29:43 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/15 14:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:29:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:29:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:29:57 Starting bucket ops to source bucket
2021/12/15 14:29:57 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:29:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/15 14:30:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 14:30:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 14:30:14 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 14:30:19 Waiting for app: TestFlexReset1 to get deployed
2021/12/15 14:30:19 App: TestFlexReset1 got deployed
2021/12/15 14:30:24 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/15 14:30:24 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 14:30:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:30:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:30:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 14:30:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 14:30:39 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 14:30:44 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/15 14:30:44 App: TestFlexReset1 got un-deployed
2021/12/15 14:30:44 No eventing-con process running
2021/12/15 14:30:44 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/15 14:30:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:30:56 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:30:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:31:07 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 14:31:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.32s)
=== RUN   TestFlexReset2
2021/12/15 14:31:12 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 14:31:12 App: TestFlexReset2 got un-deployed
2021/12/15 14:31:12 No eventing-con process running
2021/12/15 14:31:12 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/15 14:31:14 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:31:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:31:21 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:31:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:31:26 Starting bucket ops to source bucket
2021/12/15 14:31:27 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:31:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/15 14:31:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 14:31:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 14:31:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 14:31:48 Waiting for app: TestFlexReset2 to get deployed
2021/12/15 14:31:48 App: TestFlexReset2 got deployed
2021/12/15 14:31:53 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/15 14:31:53 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/15 14:31:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:31:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:31:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 14:32:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 14:32:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 14:32:13 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/15 14:32:13 App: TestFlexReset2 got un-deployed
2021/12/15 14:32:13 No eventing-con process running
2021/12/15 14:32:13 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/15 14:32:15 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:32:25 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:32:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/15 14:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.27s)
=== RUN   TestN1QLLabelledBreak
2021/12/15 14:32:42 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 14:32:42 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 14:32:42 No eventing-con process running
2021/12/15 14:32:42 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/15 14:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:32:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:32:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:32:55 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:32:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 14:33:02 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 14:33:07 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 14:33:12 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 14:33:17 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/15 14:33:17 App: TestN1QLLabelledBreak got deployed
2021/12/15 14:33:17 Starting bucket ops to source bucket
2021/12/15 14:33:26 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 14:33:31 Waiting for dst bucket item count to get to: 10000 curr count: 210
2021/12/15 14:33:36 Waiting for dst bucket item count to get to: 10000 curr count: 210
2021/12/15 14:33:41 Waiting for dst bucket item count to get to: 10000 curr count: 3545
2021/12/15 14:33:46 Waiting for dst bucket item count to get to: 10000 curr count: 3545
2021/12/15 14:33:51 Waiting for dst bucket item count to get to: 10000 curr count: 7603
2021/12/15 14:33:56 Waiting for dst bucket item count to get to: 10000 curr count: 7603
2021/12/15 14:33:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 14:33:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:33:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:34:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 14:34:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 14:34:11 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 14:34:16 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/15 14:34:16 App: TestN1QLLabelledBreak got un-deployed
2021/12/15 14:34:16 No eventing-con process running
2021/12/15 14:34:16 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/15 14:34:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:34:28 Waiting for dst bucket item count to get to: 0 curr count: 3356
2021/12/15 14:34:33 Waiting for dst bucket item count to get to: 0 curr count: 856
2021/12/15 14:34:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:34:35 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:34:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 14:34:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.19s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/15 14:34:50 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 14:34:50 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 14:34:50 No eventing-con process running
2021/12/15 14:34:50 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/15 14:34:52 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:34:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:35:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:35:03 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:35:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/15 14:35:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 14:35:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 14:35:20 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 14:35:25 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/15 14:35:25 App: TestN1QLUnlabelledBreak got deployed
2021/12/15 14:35:25 Starting bucket ops to source bucket
2021/12/15 14:35:34 Waiting for dst bucket item count to get to: 10000 curr count: 274
2021/12/15 14:35:39 Waiting for dst bucket item count to get to: 10000 curr count: 274
2021/12/15 14:35:44 Waiting for dst bucket item count to get to: 10000 curr count: 793
2021/12/15 14:35:49 Waiting for dst bucket item count to get to: 10000 curr count: 3309
2021/12/15 14:35:54 Waiting for dst bucket item count to get to: 10000 curr count: 6207
2021/12/15 14:35:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 14:35:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:35:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:35:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 14:36:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 14:36:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 14:36:14 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/15 14:36:14 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/15 14:36:14 No eventing-con process running
2021/12/15 14:36:15 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/15 14:36:16 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:36:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 14:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:36:28 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:36:38 Waiting for dst bucket item count to get to: 0 curr count: 7738
2021/12/15 14:36:43 Waiting for dst bucket item count to get to: 0 curr count: 2738
2021/12/15 14:36:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.30s)
=== RUN   TestN1QLThrowStatement
2021/12/15 14:36:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 14:36:48 App: TestN1QLThrowStatement got un-deployed
2021/12/15 14:36:48 No eventing-con process running
2021/12/15 14:36:48 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/15 14:36:50 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:36:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:36:57 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:37:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:37:02 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:37:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/15 14:37:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 14:37:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 14:37:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 14:37:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/15 14:37:23 App: TestN1QLThrowStatement got deployed
2021/12/15 14:37:23 Starting bucket ops to source bucket
2021/12/15 14:37:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 14:37:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/15 14:37:42 Waiting for dst bucket item count to get to: 10000 curr count: 971
2021/12/15 14:37:47 Waiting for dst bucket item count to get to: 10000 curr count: 2914
2021/12/15 14:37:52 Waiting for dst bucket item count to get to: 10000 curr count: 4939
2021/12/15 14:37:57 Waiting for dst bucket item count to get to: 10000 curr count: 6957
2021/12/15 14:38:02 Waiting for dst bucket item count to get to: 10000 curr count: 8931
2021/12/15 14:38:07 Waiting for dst bucket item count to get to: 10000 curr count: 9663
2021/12/15 14:38:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/15 14:38:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:38:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:38:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 14:38:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 14:38:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 14:38:28 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/15 14:38:28 App: TestN1QLThrowStatement got un-deployed
2021/12/15 14:38:28 No eventing-con process running
2021/12/15 14:38:28 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/15 14:38:30 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:38:40 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 14:38:45 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 14:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:38:46 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:38:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/15 14:38:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.14s)
=== RUN   TestN1QLNestedForLoop
2021/12/15 14:39:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 14:39:01 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 14:39:01 No eventing-con process running
2021/12/15 14:39:01 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/15 14:39:03 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:39:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:39:10 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:39:15 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:39:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/15 14:39:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 14:39:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 14:39:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 14:39:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/15 14:39:36 App: TestN1QLNestedForLoop got deployed
2021/12/15 14:39:36 Starting bucket ops to source bucket
2021/12/15 14:39:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/15 14:39:51 Waiting for dst bucket item count to get to: 5000 curr count: 143
2021/12/15 14:39:56 Waiting for dst bucket item count to get to: 5000 curr count: 143
2021/12/15 14:40:01 Waiting for dst bucket item count to get to: 5000 curr count: 2100
2021/12/15 14:40:06 Waiting for dst bucket item count to get to: 5000 curr count: 2100
2021/12/15 14:40:11 Waiting for dst bucket item count to get to: 5000 curr count: 4491
2021/12/15 14:40:16 Waiting for dst bucket item count to get to: 5000 curr count: 4491
2021/12/15 14:40:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 14:40:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:40:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:40:21 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 14:40:26 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 14:40:31 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 14:40:36 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/15 14:40:36 App: TestN1QLNestedForLoop got un-deployed
2021/12/15 14:40:36 No eventing-con process running
2021/12/15 14:40:36 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/15 14:40:38 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:40:48 Waiting for dst bucket item count to get to: 0 curr count: 2737
2021/12/15 14:40:53 Waiting for dst bucket item count to get to: 0 curr count: 237
2021/12/15 14:40:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:40:54 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:41:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 14:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.14s)
=== RUN   TestN1QLAutoClose
2021/12/15 14:41:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 14:41:10 App: TestN1QLAutoClose got un-deployed
2021/12/15 14:41:10 No eventing-con process running
2021/12/15 14:41:10 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/15 14:41:11 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:41:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:41:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:41:23 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:41:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/15 14:41:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 14:41:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 14:41:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 14:41:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/15 14:41:44 App: TestN1QLAutoClose got deployed
2021/12/15 14:41:44 Starting bucket ops to source bucket
2021/12/15 14:41:50 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/15 14:41:55 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/15 14:42:00 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/15 14:42:05 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/15 14:42:10 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/15 14:42:15 Waiting for dst bucket item count to get to: 500 curr count: 39
2021/12/15 14:42:20 Waiting for dst bucket item count to get to: 500 curr count: 51
2021/12/15 14:42:25 Waiting for dst bucket item count to get to: 500 curr count: 59
2021/12/15 14:42:30 Waiting for dst bucket item count to get to: 500 curr count: 72
2021/12/15 14:42:35 Waiting for dst bucket item count to get to: 500 curr count: 93
2021/12/15 14:42:40 Waiting for dst bucket item count to get to: 500 curr count: 93
2021/12/15 14:42:45 Waiting for dst bucket item count to get to: 500 curr count: 114
2021/12/15 14:42:50 Waiting for dst bucket item count to get to: 500 curr count: 114
2021/12/15 14:42:55 Waiting for dst bucket item count to get to: 500 curr count: 114
2021/12/15 14:43:00 Waiting for dst bucket item count to get to: 500 curr count: 135
2021/12/15 14:43:05 Waiting for dst bucket item count to get to: 500 curr count: 135
2021/12/15 14:43:10 Waiting for dst bucket item count to get to: 500 curr count: 156
2021/12/15 14:43:15 Waiting for dst bucket item count to get to: 500 curr count: 156
2021/12/15 14:43:20 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/15 14:43:25 Waiting for dst bucket item count to get to: 500 curr count: 177
2021/12/15 14:43:30 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/15 14:43:35 Waiting for dst bucket item count to get to: 500 curr count: 200
2021/12/15 14:43:40 Waiting for dst bucket item count to get to: 500 curr count: 222
2021/12/15 14:43:45 Waiting for dst bucket item count to get to: 500 curr count: 222
2021/12/15 14:43:50 Waiting for dst bucket item count to get to: 500 curr count: 243
2021/12/15 14:43:55 Waiting for dst bucket item count to get to: 500 curr count: 243
2021/12/15 14:44:00 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/15 14:44:05 Waiting for dst bucket item count to get to: 500 curr count: 263
2021/12/15 14:44:10 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/15 14:44:15 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/15 14:44:20 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/15 14:44:25 Waiting for dst bucket item count to get to: 500 curr count: 307
2021/12/15 14:44:30 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/15 14:44:35 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/15 14:44:40 Waiting for dst bucket item count to get to: 500 curr count: 349
2021/12/15 14:44:45 Waiting for dst bucket item count to get to: 500 curr count: 349
2021/12/15 14:44:50 Waiting for dst bucket item count to get to: 500 curr count: 370
2021/12/15 14:44:55 Waiting for dst bucket item count to get to: 500 curr count: 370
2021/12/15 14:45:00 Waiting for dst bucket item count to get to: 500 curr count: 391
2021/12/15 14:45:05 Waiting for dst bucket item count to get to: 500 curr count: 402
2021/12/15 14:45:10 Waiting for dst bucket item count to get to: 500 curr count: 412
2021/12/15 14:45:15 Waiting for dst bucket item count to get to: 500 curr count: 424
2021/12/15 14:45:20 Waiting for dst bucket item count to get to: 500 curr count: 432
2021/12/15 14:45:25 Waiting for dst bucket item count to get to: 500 curr count: 444
2021/12/15 14:45:30 Waiting for dst bucket item count to get to: 500 curr count: 454
2021/12/15 14:45:35 Waiting for dst bucket item count to get to: 500 curr count: 466
2021/12/15 14:45:40 Waiting for dst bucket item count to get to: 500 curr count: 476
2021/12/15 14:45:45 Waiting for dst bucket item count to get to: 500 curr count: 485
2021/12/15 14:45:50 Waiting for dst bucket item count to get to: 500 curr count: 496
2021/12/15 14:45:55 Waiting for dst bucket item count to get to: 500 curr count: 499
2021/12/15 14:45:55 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/15 14:45:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:45:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:46:00 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 14:46:06 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 14:46:11 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 14:46:16 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/15 14:46:16 App: TestN1QLAutoClose got un-deployed
2021/12/15 14:46:16 No eventing-con process running
2021/12/15 14:46:16 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/15 14:46:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:46:27 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 14:46:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:46:29 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:46:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/15 14:46:44 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/15 14:46:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.63s)
=== RUN   TestN1QLPosParams
2021/12/15 14:46:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 14:46:49 App: TestN1QLPosParams got un-deployed
2021/12/15 14:46:49 No eventing-con process running
2021/12/15 14:46:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/15 14:46:51 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:46:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:46:58 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:47:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:47:03 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:47:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/15 14:47:09 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 14:47:14 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 14:47:19 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 14:47:24 Waiting for app: TestN1QLPosParams to get deployed
2021/12/15 14:47:24 App: TestN1QLPosParams got deployed
2021/12/15 14:47:24 Starting bucket ops to source bucket
2021/12/15 14:47:34 Waiting for dst bucket item count to get to: 5000 curr count: 516
2021/12/15 14:47:39 Waiting for dst bucket item count to get to: 5000 curr count: 1500
2021/12/15 14:47:44 Waiting for dst bucket item count to get to: 5000 curr count: 3484
2021/12/15 14:47:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 14:47:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:47:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:47:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 14:47:54 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 14:47:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 14:48:04 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/15 14:48:04 App: TestN1QLPosParams got un-deployed
2021/12/15 14:48:04 No eventing-con process running
2021/12/15 14:48:04 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/15 14:48:06 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:48:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 14:48:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:48:27 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/15 14:48:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (103.36s)
=== RUN   TestN1QLExhaustConnPool
2021/12/15 14:48:32 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 14:48:33 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 14:48:33 No eventing-con process running
2021/12/15 14:48:33 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/15 14:48:34 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:48:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:48:41 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:48:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:48:46 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:48:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/15 14:48:52 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 14:48:57 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 14:49:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 14:49:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/15 14:49:07 App: TestN1QLExhaustConnPool got deployed
2021/12/15 14:49:07 Starting bucket ops to source bucket
2021/12/15 14:49:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/15 14:49:18 Waiting for dst bucket item count to get to: 100 curr count: 32
2021/12/15 14:49:23 Waiting for dst bucket item count to get to: 100 curr count: 32
2021/12/15 14:49:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/15 14:49:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:49:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:49:28 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 14:49:33 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 14:49:38 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 14:49:43 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/15 14:49:43 App: TestN1QLExhaustConnPool got un-deployed
2021/12/15 14:49:43 No eventing-con process running
2021/12/15 14:49:43 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/15 14:49:44 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:49:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 14:49:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:49:56 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:50:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/15 14:50:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.78s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/15 14:50:11 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 14:50:11 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 14:50:11 No eventing-con process running
2021/12/15 14:50:11 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/15 14:50:13 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:50:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:50:20 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:50:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:50:25 Starting bucket ops to source bucket
2021/12/15 14:50:42 Starting bucket ops to source bucket
2021/12/15 14:51:00 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 9891
2021/12/15 14:51:01 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/15 14:51:06 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 14:51:11 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 14:51:16 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 14:51:21 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/15 14:51:21 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/15 14:51:21 Starting bucket ops to source bucket
2021/12/15 14:51:26 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:31 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:36 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:41 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:46 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:51 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:51:56 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:52:01 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:52:06 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:52:11 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:52:16 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/15 14:52:16 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/15 14:52:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:52:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:52:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 14:52:27 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 14:52:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 14:52:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/15 14:52:37 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/15 14:52:37 No eventing-con process running
2021/12/15 14:52:37 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/15 14:52:39 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:52:49 Waiting for dst bucket item count to get to: 0 curr count: 25383
2021/12/15 14:52:54 Waiting for dst bucket item count to get to: 0 curr count: 10383
2021/12/15 14:52:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:52:55 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:53:05 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/15 14:53:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.05s)
=== RUN   TestN1QLQueryConsistency
2021/12/15 14:53:10 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 14:53:10 App: n1qlquery_consistency got un-deployed
2021/12/15 14:53:10 No eventing-con process running
2021/12/15 14:53:10 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/15 14:53:12 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:53:19 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:53:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:53:24 Starting bucket ops to source bucket
2021/12/15 14:53:27 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:53:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/15 14:53:33 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 14:53:38 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 14:53:43 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 14:53:48 Waiting for app: n1qlquery_consistency to get deployed
2021/12/15 14:53:48 App: n1qlquery_consistency got deployed
2021/12/15 14:53:53 Waiting for dst bucket item count to get to: 5000 curr count: 15
2021/12/15 14:53:58 Waiting for dst bucket item count to get to: 5000 curr count: 4743
2021/12/15 14:54:03 Waiting for dst bucket item count to get to: 5000 curr count: 4743
2021/12/15 14:54:03 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/15 14:54:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/15 14:54:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:54:08 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 14:54:13 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 14:54:18 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 14:54:23 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/15 14:54:23 App: n1qlquery_consistency got un-deployed
2021/12/15 14:54:23 No eventing-con process running
2021/12/15 14:54:23 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/15 14:54:25 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:54:35 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 14:54:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:54:37 Context: Bucket flush, response code: 200 dump: 
2021/12/15 14:54:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/15 14:54:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.56s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1538.492s

N1ql tests took 0 hours, 25 minutes and 43 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/15 14:56:22 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:46838->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:46838->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/15 14:56:41 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 14:56:41 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 14:56:41 No eventing-con process running
2021/12/15 14:56:41 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/15 14:56:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:56:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 14:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 14:56:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:56:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 14:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:56:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 14:56:59 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 14:57:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 14:57:05 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 14:57:10 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 14:57:15 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 14:57:20 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/15 14:57:20 App: TestDuplicateWithEventingRebIn got deployed
2021/12/15 14:57:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 14:57:20 Starting bucket ops to source bucket
2021/12/15 14:57:35 Removing node(s): [] from the cluster
2021/12/15 14:57:40 Rebalance progress: 0
2021/12/15 14:57:45 Rebalance progress: 0
2021/12/15 14:57:50 Rebalance progress: 25
2021/12/15 14:57:55 Rebalance progress: 25
2021/12/15 14:58:00 Rebalance progress: 25
2021/12/15 14:58:05 Rebalance progress: 25
2021/12/15 14:58:10 Rebalance progress: 25
2021/12/15 14:58:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:58:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:58:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 14:58:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 14:58:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 14:58:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 14:58:20 Rebalance progress: 25
2021/12/15 14:58:25 Rebalance progress: 25
2021/12/15 14:58:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:58:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 14:58:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 14:58:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 14:58:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 14:58:30 Waiting for src bucket item count to get to: 200000 curr count: 96471
2021/12/15 14:58:35 Waiting for src bucket item count to get to: 200000 curr count: 113055
2021/12/15 14:58:40 Waiting for src bucket item count to get to: 200000 curr count: 113055
2021/12/15 14:58:45 Waiting for src bucket item count to get to: 200000 curr count: 129681
2021/12/15 14:58:50 Waiting for src bucket item count to get to: 200000 curr count: 129681
2021/12/15 14:58:55 Waiting for src bucket item count to get to: 200000 curr count: 146169
2021/12/15 14:59:00 Waiting for src bucket item count to get to: 200000 curr count: 146169
2021/12/15 14:59:05 Waiting for src bucket item count to get to: 200000 curr count: 162638
2021/12/15 14:59:10 Waiting for src bucket item count to get to: 200000 curr count: 162638
2021/12/15 14:59:15 Waiting for src bucket item count to get to: 200000 curr count: 179231
2021/12/15 14:59:20 Waiting for src bucket item count to get to: 200000 curr count: 179231
2021/12/15 14:59:25 Waiting for src bucket item count to get to: 200000 curr count: 195748
2021/12/15 14:59:30 Waiting for src bucket item count to get to: 200000 curr count: 195748
2021/12/15 14:59:35 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 14:59:35 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 14:59:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 14:59:40 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 14:59:45 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 14:59:50 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 14:59:55 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/15 14:59:55 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/15 14:59:55 No eventing-con process running
2021/12/15 14:59:55 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/15 14:59:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:00:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:00:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:00:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:00:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:00:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (227.79s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/15 15:00:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:00:24 Removing node(s): [] from the cluster
2021/12/15 15:00:29 Rebalance progress: 100
2021/12/15 15:00:34 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 15:00:34 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 15:00:34 No eventing-con process running
2021/12/15 15:00:34 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/15 15:00:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:00:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:00:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:00:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:00:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:00:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:00:52 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:00:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 15:00:58 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 15:01:03 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 15:01:08 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 15:01:13 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/15 15:01:13 App: TestDuplicateWithEventingRebOut got deployed
2021/12/15 15:01:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:01:13 Starting bucket ops to source bucket
2021/12/15 15:01:18 Rebalance progress: 25
2021/12/15 15:01:23 Rebalance progress: 25
2021/12/15 15:01:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:01:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:01:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 15:01:28 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:01:28 Waiting for src bucket item count to get to: 200000 curr count: 9344
2021/12/15 15:01:33 Waiting for src bucket item count to get to: 200000 curr count: 25605
2021/12/15 15:01:38 Waiting for src bucket item count to get to: 200000 curr count: 25605
2021/12/15 15:01:43 Waiting for src bucket item count to get to: 200000 curr count: 41997
2021/12/15 15:01:48 Waiting for src bucket item count to get to: 200000 curr count: 41997
2021/12/15 15:01:53 Waiting for src bucket item count to get to: 200000 curr count: 58721
2021/12/15 15:01:58 Waiting for src bucket item count to get to: 200000 curr count: 58721
2021/12/15 15:02:03 Waiting for src bucket item count to get to: 200000 curr count: 58721
2021/12/15 15:02:08 Waiting for src bucket item count to get to: 200000 curr count: 75225
2021/12/15 15:02:13 Waiting for src bucket item count to get to: 200000 curr count: 75225
2021/12/15 15:02:19 Waiting for src bucket item count to get to: 200000 curr count: 90837
2021/12/15 15:02:24 Waiting for src bucket item count to get to: 200000 curr count: 90837
2021/12/15 15:02:29 Waiting for src bucket item count to get to: 200000 curr count: 107313
2021/12/15 15:02:34 Waiting for src bucket item count to get to: 200000 curr count: 107313
2021/12/15 15:02:39 Waiting for src bucket item count to get to: 200000 curr count: 123688
2021/12/15 15:02:44 Waiting for src bucket item count to get to: 200000 curr count: 123688
2021/12/15 15:02:49 Waiting for src bucket item count to get to: 200000 curr count: 140215
2021/12/15 15:02:54 Waiting for src bucket item count to get to: 200000 curr count: 140215
2021/12/15 15:02:59 Waiting for src bucket item count to get to: 200000 curr count: 156823
2021/12/15 15:03:04 Waiting for src bucket item count to get to: 200000 curr count: 156823
2021/12/15 15:03:09 Waiting for src bucket item count to get to: 200000 curr count: 172957
2021/12/15 15:03:14 Waiting for src bucket item count to get to: 200000 curr count: 189061
2021/12/15 15:03:19 Waiting for src bucket item count to get to: 200000 curr count: 189061
2021/12/15 15:03:24 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 15:03:24 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 15:03:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:03:29 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 15:03:34 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 15:03:39 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 15:03:44 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/15 15:03:44 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/15 15:03:44 No eventing-con process running
2021/12/15 15:03:44 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/15 15:03:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:03:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:03:55 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:03:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:03:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:04:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:04:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.76s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/15 15:04:17 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 15:04:17 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 15:04:17 No eventing-con process running
2021/12/15 15:04:17 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/15 15:04:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:04:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:04:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:04:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:04:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:04:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:04:36 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:04:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 15:04:42 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 15:04:47 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 15:04:52 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 15:04:57 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/15 15:04:57 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/15 15:04:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:04:57 Starting bucket ops to source bucket
2021/12/15 15:04:57 Removing node(s): [] from the cluster
2021/12/15 15:05:02 Rebalance progress: 25
2021/12/15 15:05:07 Rebalance progress: 25
2021/12/15 15:05:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:05:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/15 15:05:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:05:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:05:19 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:05:24 Rebalance progress: 16.66666666666667
2021/12/15 15:05:29 Rebalance progress: 16.66666666666667
2021/12/15 15:05:34 Rebalance progress: 16.66666666666667
2021/12/15 15:05:39 Rebalance progress: 16.66666666666667
2021/12/15 15:05:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:05:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:05:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:05:49 Rebalance progress: 16.66666666666667
2021/12/15 15:05:54 Rebalance progress: 16.66666666666667
2021/12/15 15:05:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:05:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:05:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:05:59 Waiting for src bucket item count to get to: 200000 curr count: 78067
2021/12/15 15:06:04 Waiting for src bucket item count to get to: 200000 curr count: 94139
2021/12/15 15:06:09 Waiting for src bucket item count to get to: 200000 curr count: 94139
2021/12/15 15:06:14 Waiting for src bucket item count to get to: 200000 curr count: 109989
2021/12/15 15:06:19 Waiting for src bucket item count to get to: 200000 curr count: 109989
2021/12/15 15:06:24 Waiting for src bucket item count to get to: 200000 curr count: 126188
2021/12/15 15:06:29 Waiting for src bucket item count to get to: 200000 curr count: 126188
2021/12/15 15:06:34 Waiting for src bucket item count to get to: 200000 curr count: 142285
2021/12/15 15:06:39 Waiting for src bucket item count to get to: 200000 curr count: 142285
2021/12/15 15:06:44 Waiting for src bucket item count to get to: 200000 curr count: 158502
2021/12/15 15:06:49 Waiting for src bucket item count to get to: 200000 curr count: 158502
2021/12/15 15:06:54 Waiting for src bucket item count to get to: 200000 curr count: 174273
2021/12/15 15:06:59 Waiting for src bucket item count to get to: 200000 curr count: 174273
2021/12/15 15:07:04 Waiting for src bucket item count to get to: 200000 curr count: 190695
2021/12/15 15:07:09 Waiting for src bucket item count to get to: 200000 curr count: 190695
2021/12/15 15:07:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 15:07:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 15:07:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:07:19 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 15:07:24 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 15:07:29 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 15:07:34 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/15 15:07:34 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/15 15:07:34 No eventing-con process running
2021/12/15 15:07:34 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/15 15:07:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:07:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:07:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:07:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:07:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:07:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.75s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/15 15:08:08 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 15:08:08 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 15:08:08 No eventing-con process running
2021/12/15 15:08:08 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/15 15:08:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:08:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:08:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:08:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:08:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:08:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:08:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:08:27 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:08:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 15:08:33 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 15:08:38 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 15:08:43 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 15:08:48 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/15 15:08:48 App: TestDuplicateWithKVRebIn got deployed
2021/12/15 15:08:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 15:08:48 Starting bucket ops to source bucket
2021/12/15 15:08:48 Removing node(s): [] from the cluster
2021/12/15 15:08:53 Rebalance progress: 16.66666666666667
2021/12/15 15:08:58 Rebalance progress: 16.66666666666667
2021/12/15 15:09:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:09:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:09:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:09:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:09:08 Rebalance progress: 16.66666666666667
2021/12/15 15:09:13 Rebalance progress: 16.66666666666667
2021/12/15 15:09:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:09:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:09:18 Waiting for src bucket item count to get to: 200000 curr count: 33681
2021/12/15 15:09:23 Waiting for src bucket item count to get to: 200000 curr count: 49758
2021/12/15 15:09:28 Waiting for src bucket item count to get to: 200000 curr count: 49758
2021/12/15 15:09:33 Waiting for src bucket item count to get to: 200000 curr count: 66242
2021/12/15 15:09:38 Waiting for src bucket item count to get to: 200000 curr count: 66242
2021/12/15 15:09:43 Waiting for src bucket item count to get to: 200000 curr count: 81562
2021/12/15 15:09:48 Waiting for src bucket item count to get to: 200000 curr count: 81562
2021/12/15 15:09:53 Waiting for src bucket item count to get to: 200000 curr count: 81562
2021/12/15 15:09:58 Waiting for src bucket item count to get to: 200000 curr count: 97264
2021/12/15 15:10:03 Waiting for src bucket item count to get to: 200000 curr count: 97264
2021/12/15 15:10:08 Waiting for src bucket item count to get to: 200000 curr count: 113383
2021/12/15 15:10:13 Waiting for src bucket item count to get to: 200000 curr count: 113383
2021/12/15 15:10:18 Waiting for src bucket item count to get to: 200000 curr count: 129675
2021/12/15 15:10:23 Waiting for src bucket item count to get to: 200000 curr count: 129675
2021/12/15 15:10:28 Waiting for src bucket item count to get to: 200000 curr count: 145599
2021/12/15 15:10:33 Waiting for src bucket item count to get to: 200000 curr count: 145599
2021/12/15 15:10:38 Waiting for src bucket item count to get to: 200000 curr count: 161697
2021/12/15 15:10:43 Waiting for src bucket item count to get to: 200000 curr count: 161697
2021/12/15 15:10:48 Waiting for src bucket item count to get to: 200000 curr count: 177174
2021/12/15 15:10:53 Waiting for src bucket item count to get to: 200000 curr count: 177174
2021/12/15 15:10:58 Waiting for src bucket item count to get to: 200000 curr count: 193405
2021/12/15 15:11:03 Waiting for src bucket item count to get to: 200000 curr count: 193405
2021/12/15 15:11:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 15:11:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 15:11:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:11:13 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 15:11:18 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 15:11:23 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 15:11:28 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/15 15:11:28 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/15 15:11:28 No eventing-con process running
2021/12/15 15:11:28 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/15 15:11:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:11:40 Waiting for dst bucket item count to get to: 0 curr count: 44881
2021/12/15 15:11:45 Waiting for dst bucket item count to get to: 0 curr count: 44881
2021/12/15 15:11:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:11:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:11:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:11:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:11:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (238.96s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/15 15:12:02 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 15:12:02 Removing node(s): [] from the cluster
2021/12/15 15:12:07 Rebalance progress: 100
2021/12/15 15:12:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 15:12:12 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 15:12:12 No eventing-con process running
2021/12/15 15:12:12 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/15 15:12:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:12:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:12:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:12:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:12:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:12:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:12:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:12:31 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:12:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 15:12:37 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 15:12:42 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 15:12:47 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 15:12:52 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/15 15:12:52 App: TestDuplicateWithKVRebOut got deployed
2021/12/15 15:12:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:12:52 Starting bucket ops to source bucket
2021/12/15 15:12:57 Rebalance progress: 16.66666666666667
2021/12/15 15:13:02 Rebalance progress: 16.66666666666667
2021/12/15 15:13:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:13:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:13:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:13:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:13:07 Waiting for src bucket item count to get to: 200000 curr count: 11325
2021/12/15 15:13:12 Waiting for src bucket item count to get to: 200000 curr count: 27249
2021/12/15 15:13:17 Waiting for src bucket item count to get to: 200000 curr count: 27249
2021/12/15 15:13:22 Waiting for src bucket item count to get to: 200000 curr count: 43451
2021/12/15 15:13:27 Waiting for src bucket item count to get to: 200000 curr count: 43451
2021/12/15 15:13:32 Waiting for src bucket item count to get to: 200000 curr count: 59414
2021/12/15 15:13:37 Waiting for src bucket item count to get to: 200000 curr count: 59414
2021/12/15 15:13:42 Waiting for src bucket item count to get to: 200000 curr count: 59414
2021/12/15 15:13:47 Waiting for src bucket item count to get to: 200000 curr count: 74729
2021/12/15 15:13:52 Waiting for src bucket item count to get to: 200000 curr count: 74729
2021/12/15 15:13:57 Waiting for src bucket item count to get to: 200000 curr count: 90309
2021/12/15 15:14:02 Waiting for src bucket item count to get to: 200000 curr count: 90309
2021/12/15 15:14:07 Waiting for src bucket item count to get to: 200000 curr count: 106434
2021/12/15 15:14:12 Waiting for src bucket item count to get to: 200000 curr count: 106434
2021/12/15 15:14:17 Waiting for src bucket item count to get to: 200000 curr count: 122515
2021/12/15 15:14:22 Waiting for src bucket item count to get to: 200000 curr count: 122515
2021/12/15 15:14:27 Waiting for src bucket item count to get to: 200000 curr count: 138471
2021/12/15 15:14:32 Waiting for src bucket item count to get to: 200000 curr count: 138471
2021/12/15 15:14:37 Waiting for src bucket item count to get to: 200000 curr count: 154489
2021/12/15 15:14:42 Waiting for src bucket item count to get to: 200000 curr count: 154489
2021/12/15 15:14:47 Waiting for src bucket item count to get to: 200000 curr count: 169733
2021/12/15 15:14:52 Waiting for src bucket item count to get to: 200000 curr count: 169733
2021/12/15 15:14:57 Waiting for src bucket item count to get to: 200000 curr count: 185449
2021/12/15 15:15:02 Waiting for src bucket item count to get to: 200000 curr count: 185449
2021/12/15 15:15:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 15:15:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 15:15:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:15:12 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 15:15:17 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 15:15:22 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 15:15:27 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/15 15:15:27 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/15 15:15:27 No eventing-con process running
2021/12/15 15:15:27 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/15 15:15:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:15:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:15:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:15:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:15:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:15:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:15:51 Waiting for dst bucket item count to get to: 0 curr count: 148166
2021/12/15 15:15:56 Waiting for dst bucket item count to get to: 0 curr count: 148166
2021/12/15 15:15:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (238.80s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/15 15:16:06 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 15:16:06 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 15:16:06 No eventing-con process running
2021/12/15 15:16:06 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/15 15:16:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:16:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:16:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:16:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:16:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:16:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:16:24 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:16:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 15:16:30 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 15:16:35 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 15:16:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 15:16:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/15 15:16:45 App: TestDuplicateWithKVSwapReb got deployed
2021/12/15 15:16:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 15:16:45 Starting bucket ops to source bucket
2021/12/15 15:16:46 Removing node(s): [] from the cluster
2021/12/15 15:16:51 Rebalance progress: 16.66666666666667
2021/12/15 15:16:56 Rebalance progress: 16.66666666666667
2021/12/15 15:17:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:17:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:17:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 15:17:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:17:07 Rebalance progress: 16.66666666666667
2021/12/15 15:17:12 Rebalance progress: 16.66666666666667
2021/12/15 15:17:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:17:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:17:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:17:22 Rebalance progress: 16.66666666666667
2021/12/15 15:17:27 Rebalance progress: 16.66666666666667
2021/12/15 15:17:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:17:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:17:32 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:17:32 Waiting for src bucket item count to get to: 200000 curr count: 69054
2021/12/15 15:17:37 Waiting for src bucket item count to get to: 200000 curr count: 69054
2021/12/15 15:17:42 Waiting for src bucket item count to get to: 200000 curr count: 84293
2021/12/15 15:17:47 Waiting for src bucket item count to get to: 200000 curr count: 84293
2021/12/15 15:17:52 Waiting for src bucket item count to get to: 200000 curr count: 100755
2021/12/15 15:17:57 Waiting for src bucket item count to get to: 200000 curr count: 100755
2021/12/15 15:18:02 Waiting for src bucket item count to get to: 200000 curr count: 117168
2021/12/15 15:18:07 Waiting for src bucket item count to get to: 200000 curr count: 117168
2021/12/15 15:18:12 Waiting for src bucket item count to get to: 200000 curr count: 133630
2021/12/15 15:18:17 Waiting for src bucket item count to get to: 200000 curr count: 133630
2021/12/15 15:18:22 Waiting for src bucket item count to get to: 200000 curr count: 149881
2021/12/15 15:18:27 Waiting for src bucket item count to get to: 200000 curr count: 149881
2021/12/15 15:18:32 Waiting for src bucket item count to get to: 200000 curr count: 166273
2021/12/15 15:18:37 Waiting for src bucket item count to get to: 200000 curr count: 166273
2021/12/15 15:18:42 Waiting for src bucket item count to get to: 200000 curr count: 181777
2021/12/15 15:18:47 Waiting for src bucket item count to get to: 200000 curr count: 181777
2021/12/15 15:18:52 Waiting for src bucket item count to get to: 200000 curr count: 197911
2021/12/15 15:18:57 Waiting for src bucket item count to get to: 200000 curr count: 197911
2021/12/15 15:19:02 Waiting for src bucket item count to get to: 200000 curr count: 197911
2021/12/15 15:19:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 15:19:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 15:19:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:19:12 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 15:19:17 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 15:19:22 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 15:19:27 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/15 15:19:27 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/15 15:19:27 No eventing-con process running
2021/12/15 15:19:27 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/15 15:19:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:19:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:19:39 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/15 15:19:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:19:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:19:51 Waiting for dst bucket item count to get to: 0 curr count: 138496
2021/12/15 15:19:56 Waiting for dst bucket item count to get to: 0 curr count: 138496
2021/12/15 15:19:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.92s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/15 15:20:06 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:20:06 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:20:06 No eventing-con process running
2021/12/15 15:20:06 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 15:20:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:20:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:20:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:20:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:20:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:20:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:20:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:20:24 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:20:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 15:20:30 Starting bucket ops to source bucket
2021/12/15 15:20:35 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:20:40 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:20:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:20:45 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:20:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:20:45 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:20:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:20:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:20:51 Rebalance progress: 16.66666666666667
2021/12/15 15:20:56 Rebalance progress: 16.66666666666667
2021/12/15 15:21:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:21:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:21:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:21:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:21:06 Rebalance progress: 16.66666666666667
2021/12/15 15:21:11 Rebalance progress: 16.66666666666667
2021/12/15 15:21:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:21:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/15 15:21:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 15:21:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:21:27 Rebalance progress: 12.5
2021/12/15 15:21:32 Rebalance progress: 12.5
2021/12/15 15:21:37 Rebalance progress: 12.5
2021/12/15 15:21:42 Rebalance progress: 65.85365853658536
2021/12/15 15:21:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:21:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:21:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:21:52 Rebalance progress: 12.5
2021/12/15 15:21:57 Rebalance progress: 12.5
2021/12/15 15:22:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:22:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:22:07 Rebalance progress: 12.5
2021/12/15 15:22:12 Rebalance progress: 12.5
2021/12/15 15:22:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:22:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 15:22:22 Rebalance progress: 12.5
2021/12/15 15:22:27 Rebalance progress: 12.5
2021/12/15 15:22:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:22:32 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:22:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:22:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:22:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:22:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:22:52 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:22:52 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:22:52 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 15:22:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:22:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:23:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12149
2021/12/15 15:23:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12149
2021/12/15 15:23:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:23:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:23:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:23:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8450
2021/12/15 15:23:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8450
2021/12/15 15:23:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (210.04s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/15 15:23:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 15:23:36 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 15:23:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:23:36 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/15 15:23:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:23:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:23:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:23:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:23:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:23:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:23:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:23:54 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:23:55 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/15 15:24:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:24:05 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 15:24:10 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 15:24:15 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/15 15:24:15 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:24:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:24:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:24:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 15:24:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:24:22 Rebalance progress: 12.5
2021/12/15 15:24:27 Rebalance progress: 12.5
2021/12/15 15:24:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:24:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 15:24:37 Rebalance progress: 12.5
2021/12/15 15:24:42 Rebalance progress: 12.5
2021/12/15 15:24:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:24:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:24:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:24:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 15:24:57 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 15:25:02 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 15:25:07 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/15 15:25:07 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/15 15:25:07 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:25:07 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/15 15:25:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:25:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:25:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4662
2021/12/15 15:25:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:25:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:25:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:25:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4512
2021/12/15 15:25:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4512
2021/12/15 15:25:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (130.30s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/15 15:25:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 15:25:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 15:25:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:25:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/15 15:25:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:25:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:25:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:25:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:26:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:26:05 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:26:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/15 15:26:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:26:16 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 15:26:21 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 15:26:26 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/15 15:26:26 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:26:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:26:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:26:31 Rebalance progress: 12.5
2021/12/15 15:26:36 Rebalance progress: 12.5
2021/12/15 15:26:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:26:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:26:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:26:47 Rebalance progress: 12.5
2021/12/15 15:26:52 Rebalance progress: 12.5
2021/12/15 15:26:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:26:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:26:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 15:26:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:27:02 Rebalance progress: 12.5
2021/12/15 15:27:07 Rebalance progress: 12.5
2021/12/15 15:27:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:27:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:27:18 Rebalance progress: 12.5
2021/12/15 15:27:23 Rebalance progress: 12.5
2021/12/15 15:27:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:27:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:27:33 Rebalance progress: 12.5
2021/12/15 15:27:38 Rebalance progress: 12.5
2021/12/15 15:27:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:27:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 15:27:48 Rebalance progress: 12.5
2021/12/15 15:27:53 Rebalance progress: 12.5
2021/12/15 15:27:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:27:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:27:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:28:03 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 15:28:08 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 15:28:13 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 15:28:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/15 15:28:18 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/15 15:28:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:28:18 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/15 15:28:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:28:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:28:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:28:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:28:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:28:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10695
2021/12/15 15:28:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (180.59s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/15 15:28:47 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 15:28:47 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 15:28:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:28:47 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/15 15:28:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:28:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:28:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:28:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:29:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:29:00 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:29:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 15:29:06 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:29:11 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 15:29:16 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 15:29:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/15 15:29:21 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:29:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:29:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:29:27 Rebalance progress: 12.5
2021/12/15 15:29:32 Rebalance progress: 12.5
2021/12/15 15:29:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:29:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:29:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:29:42 Rebalance progress: 12.5
2021/12/15 15:29:47 Rebalance progress: 12.5
2021/12/15 15:29:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:29:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:29:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:29:57 Rebalance progress: 12.5
2021/12/15 15:30:02 Rebalance progress: 12.5
2021/12/15 15:30:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:30:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:30:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:30:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 15:30:17 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 15:30:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 15:30:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/15 15:30:27 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/15 15:30:27 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:30:27 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/15 15:30:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:30:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:30:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/15 15:30:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:30:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:30:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:30:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/15 15:30:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6101
2021/12/15 15:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.51s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/15 15:31:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:31:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:31:06 Rebalance progress: 100
2021/12/15 15:31:11 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 15:31:11 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 15:31:11 No eventing-con process running
2021/12/15 15:31:11 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/15 15:31:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:31:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:31:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:31:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:31:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:31:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:31:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:31:25 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:31:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 15:31:31 Starting bucket ops to source bucket
2021/12/15 15:31:36 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 15:31:41 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 15:31:46 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/15 15:31:46 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:31:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:31:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:31:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:31:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:31:51 Rebalance progress: 12.5
2021/12/15 15:31:56 Rebalance progress: 12.5
2021/12/15 15:32:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:32:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 15:32:02 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 15:32:07 Rebalance progress: 12.5
2021/12/15 15:32:12 Rebalance progress: 12.5
2021/12/15 15:32:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:32:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:32:22 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 15:32:27 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 15:32:32 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 15:32:37 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/15 15:32:37 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/15 15:32:37 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:32:37 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/15 15:32:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:32:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4541
2021/12/15 15:32:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:32:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:32:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:33:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1980
2021/12/15 15:33:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1980
2021/12/15 15:33:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.27s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/15 15:33:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:33:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:33:16 Rebalance progress: 100
2021/12/15 15:33:21 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 15:33:21 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 15:33:21 No eventing-con process running
2021/12/15 15:33:21 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/15 15:33:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:33:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:33:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:33:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:33:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:33:34 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/15 15:33:40 Starting bucket ops to source bucket
2021/12/15 15:33:45 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 15:33:50 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 15:33:55 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/15 15:33:55 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:33:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:33:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:33:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:33:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:34:01 Rebalance progress: 12.5
2021/12/15 15:34:06 Rebalance progress: 12.5
2021/12/15 15:34:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:34:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:34:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:34:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 15:34:11 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 15:34:16 Rebalance progress: 12.5
2021/12/15 15:34:21 Rebalance progress: 12.5
2021/12/15 15:34:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:34:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:34:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:34:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:34:26 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/15 15:34:31 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 15:34:36 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 15:34:41 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 15:34:46 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/15 15:34:46 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/15 15:34:46 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:34:46 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/15 15:34:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:34:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:34:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 36985
2021/12/15 15:34:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:35:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:35:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:35:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.27s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/15 15:35:15 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 15:35:15 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 15:35:15 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:35:15 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/15 15:35:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:35:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:35:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:35:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:35:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:35:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:35:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:35:33 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:35:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:35:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:35:34 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:35:35 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 15:35:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:35:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 15:35:51 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 15:35:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/15 15:35:56 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/15 15:36:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 15:36:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 15:36:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/15 15:36:11 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:36:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:36:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:36:16 Rebalance progress: 12.5
2021/12/15 15:36:21 Rebalance progress: 12.5
2021/12/15 15:36:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:36:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:36:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:36:31 Rebalance progress: 12.5
2021/12/15 15:36:36 Rebalance progress: 12.5
2021/12/15 15:36:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:36:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 15:36:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:36:47 Rebalance progress: 12.5
2021/12/15 15:36:52 Rebalance progress: 12.5
2021/12/15 15:36:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:36:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:36:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:37:02 Rebalance progress: 12.5
2021/12/15 15:37:07 Rebalance progress: 12.5
2021/12/15 15:37:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:37:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:37:17 Rebalance progress: 12.5
2021/12/15 15:37:22 Rebalance progress: 12.5
2021/12/15 15:37:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:37:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 15:37:32 Rebalance progress: 12.5
2021/12/15 15:37:37 Rebalance progress: 12.5
2021/12/15 15:37:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:37:42 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:37:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 15:37:52 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 15:37:57 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 15:38:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/15 15:38:02 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/15 15:38:02 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:38:02 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/15 15:38:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:38:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:38:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12155
2021/12/15 15:38:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:38:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:38:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:38:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12155
2021/12/15 15:38:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:38:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:38:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 15:38:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 15:38:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 15:38:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/15 15:38:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/15 15:38:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:38:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/15 15:38:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:38:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:38:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:38:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:39:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:39:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (236.11s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/15 15:39:11 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 15:39:11 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 15:39:11 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:39:11 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/15 15:39:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:39:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:39:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:39:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:39:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:39:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:39:29 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:39:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:39:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:39:30 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:39:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 15:39:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:39:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 15:39:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 15:39:52 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/15 15:39:52 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/15 15:39:57 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 15:40:02 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 15:40:07 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/15 15:40:07 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:40:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:40:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:40:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/15 15:40:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:40:13 Rebalance progress: 12.5
2021/12/15 15:40:18 Rebalance progress: 12.5
2021/12/15 15:40:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:40:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 15:40:28 Rebalance progress: 12.5
2021/12/15 15:40:33 Rebalance progress: 12.5
2021/12/15 15:40:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:40:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:40:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:40:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 15:40:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 15:40:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 15:40:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/15 15:40:59 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/15 15:40:59 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:40:59 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/15 15:41:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:41:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:41:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4434
2021/12/15 15:41:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4434
2021/12/15 15:41:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:41:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:41:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:41:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6179
2021/12/15 15:41:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:41:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:41:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 15:41:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 15:41:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 15:41:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/15 15:41:49 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/15 15:41:49 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:41:49 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/15 15:41:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:41:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:41:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:41:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.66s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/15 15:42:12 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 15:42:12 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 15:42:12 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:42:12 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/15 15:42:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:42:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:42:31 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 15:42:31 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 15:42:31 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:42:31 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/15 15:42:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:42:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:45 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:42:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:42:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:42:46 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:42:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 15:42:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:42:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 15:43:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 15:43:07 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/15 15:43:07 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/15 15:43:12 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 15:43:17 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 15:43:22 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/15 15:43:22 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:43:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:43:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:43:28 Rebalance progress: 12.5
2021/12/15 15:43:33 Rebalance progress: 12.5
2021/12/15 15:43:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:43:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:43:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:43:43 Rebalance progress: 12.5
2021/12/15 15:43:48 Rebalance progress: 12.5
2021/12/15 15:43:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:43:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:43:58 Rebalance progress: 12.5
2021/12/15 15:44:03 Rebalance progress: 12.5
2021/12/15 15:44:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:44:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:44:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:44:13 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 15:44:18 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 15:44:23 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 15:44:28 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/15 15:44:28 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/15 15:44:28 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:44:28 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/15 15:44:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:44:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:44:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5089
2021/12/15 15:44:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5089
2021/12/15 15:44:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:44:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:44:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:44:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7620
2021/12/15 15:44:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:44:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:45:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 15:45:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 15:45:14 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 15:45:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/15 15:45:19 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/15 15:45:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:45:19 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/15 15:45:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:45:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:45:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:45:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:45:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:45:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (209.97s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 15:45:42 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:45:42 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:45:42 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:45:42 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 15:45:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:45:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:45:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:45:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:45:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:45:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:46:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:46:01 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:46:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 15:46:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:46:12 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:46:17 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:46:22 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:46:22 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:46:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:46:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:46:43 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:46:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:46:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:47:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:47:19 Rebalance progress: 12.5
2021/12/15 15:47:24 Rebalance progress: 12.5
2021/12/15 15:47:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:47:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:47:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:47:34 Rebalance progress: 12.5
2021/12/15 15:47:39 Rebalance progress: 12.5
2021/12/15 15:47:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:47:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:47:49 Rebalance progress: 12.5
2021/12/15 15:47:54 Rebalance progress: 12.5
2021/12/15 15:47:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:47:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:48:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:48:05 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:48:10 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:48:15 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:48:20 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:48:20 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:48:20 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:48:20 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 15:48:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:48:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:48:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11216
2021/12/15 15:48:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11216
2021/12/15 15:48:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:48:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:48:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:48:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11216
2021/12/15 15:48:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.83s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 15:48:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:48:58 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:48:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:48:58 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/15 15:49:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:49:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:49:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:49:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:49:17 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:49:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/15 15:49:23 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:49:28 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:49:33 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:49:38 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/15 15:49:38 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:49:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:49:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:49:58 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:49:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:50:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:50:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:50:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:50:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:50:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:50:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:51:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:51:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:51:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:51:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:51:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:51:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:51:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:52:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:52:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:52:18 Rebalance progress: 12.5
2021/12/15 15:52:23 Rebalance progress: 12.5
2021/12/15 15:52:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:52:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:52:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:52:33 Rebalance progress: 12.5
2021/12/15 15:52:38 Rebalance progress: 12.5
2021/12/15 15:52:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:52:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:52:48 Rebalance progress: 12.5
2021/12/15 15:52:53 Rebalance progress: 12.5
2021/12/15 15:52:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:52:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:52:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:53:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:53:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:53:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:53:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/15 15:53:18 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/15 15:53:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:53:18 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/15 15:53:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:53:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:53:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4786
2021/12/15 15:53:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4786
2021/12/15 15:53:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:53:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:53:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:53:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21517
2021/12/15 15:53:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.78s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/15 15:53:57 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 15:53:57 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 15:53:57 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:53:57 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/15 15:53:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:54:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:54:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:54:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:54:16 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:54:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 15:54:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:54:27 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 15:54:32 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 15:54:37 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/15 15:54:37 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:54:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:54:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:54:42 Rebalance progress: 12.5
2021/12/15 15:54:47 Rebalance progress: 12.5
2021/12/15 15:54:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:54:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:54:52 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:55:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:55:07 Rebalance progress: 12.5
2021/12/15 15:55:12 Rebalance progress: 12.5
2021/12/15 15:55:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:55:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 15:55:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:55:23 Rebalance progress: 12.5
2021/12/15 15:55:28 Rebalance progress: 12.5
2021/12/15 15:55:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:55:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:55:38 Rebalance progress: 12.5
2021/12/15 15:55:43 Rebalance progress: 12.5
2021/12/15 15:55:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:55:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:55:48 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:55:53 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 15:55:58 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 15:56:03 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 15:56:08 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/15 15:56:08 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/15 15:56:08 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:56:08 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/15 15:56:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:56:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:56:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:56:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:56:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:56:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8587
2021/12/15 15:56:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (159.35s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/15 15:56:36 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 15:56:36 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 15:56:36 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:56:36 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/15 15:56:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:56:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:56:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:56:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:56:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:56:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:56:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:56:55 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:56:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/15 15:57:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:57:06 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 15:57:11 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 15:57:16 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/15 15:57:16 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:57:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/15 15:57:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:57:21 Rebalance progress: 12.5
2021/12/15 15:57:26 Rebalance progress: 12.5
2021/12/15 15:57:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:57:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/15 15:57:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 15:57:37 Rebalance progress: 12.5
2021/12/15 15:57:42 Rebalance progress: 12.5
2021/12/15 15:57:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:57:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:57:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 15:57:52 Rebalance progress: 12.5
2021/12/15 15:57:57 Rebalance progress: 12.5
2021/12/15 15:58:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:58:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:58:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 15:58:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 15:58:07 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 15:58:12 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 15:58:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 15:58:22 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/15 15:58:22 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/15 15:58:22 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:58:22 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/15 15:58:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:58:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:58:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6065
2021/12/15 15:58:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6065
2021/12/15 15:58:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:58:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:58:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:58:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3429
2021/12/15 15:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3429
2021/12/15 15:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (149.18s)
=== RUN   TestEventingRebInWithOps
2021/12/15 15:59:06 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 15:59:06 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/15 15:59:06 eventingSetup.go:46: No eventing-con process running
2021/12/15 15:59:06 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/15 15:59:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:59:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:59:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 15:59:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:59:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:59:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 15:59:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 15:59:24 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 15:59:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 15:59:30 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 15:59:35 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 15:59:40 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 15:59:45 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/15 15:59:45 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/15 15:59:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 15:59:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 15:59:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 15:59:51 Rebalance progress: 12.5
2021/12/15 15:59:56 Rebalance progress: 12.5
2021/12/15 16:00:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:00:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:00:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:00:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:00:06 Rebalance progress: 12.5
2021/12/15 16:00:11 Rebalance progress: 12.5
2021/12/15 16:00:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:00:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:00:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:00:16 Waiting for src bucket item count to get to: 200000 curr count: 36525
2021/12/15 16:00:21 Waiting for src bucket item count to get to: 200000 curr count: 52165
2021/12/15 16:00:26 Waiting for src bucket item count to get to: 200000 curr count: 52165
2021/12/15 16:00:31 Waiting for src bucket item count to get to: 200000 curr count: 67825
2021/12/15 16:00:36 Waiting for src bucket item count to get to: 200000 curr count: 67825
2021/12/15 16:00:41 Waiting for src bucket item count to get to: 200000 curr count: 82269
2021/12/15 16:00:46 Waiting for src bucket item count to get to: 200000 curr count: 82269
2021/12/15 16:00:51 Waiting for src bucket item count to get to: 200000 curr count: 98138
2021/12/15 16:00:56 Waiting for src bucket item count to get to: 200000 curr count: 98138
2021/12/15 16:01:01 Waiting for src bucket item count to get to: 200000 curr count: 113989
2021/12/15 16:01:06 Waiting for src bucket item count to get to: 200000 curr count: 113989
2021/12/15 16:01:11 Waiting for src bucket item count to get to: 200000 curr count: 129924
2021/12/15 16:01:16 Waiting for src bucket item count to get to: 200000 curr count: 129924
2021/12/15 16:01:21 Waiting for src bucket item count to get to: 200000 curr count: 145843
2021/12/15 16:01:26 Waiting for src bucket item count to get to: 200000 curr count: 145843
2021/12/15 16:01:31 Waiting for src bucket item count to get to: 200000 curr count: 145843
2021/12/15 16:01:36 Waiting for src bucket item count to get to: 200000 curr count: 161552
2021/12/15 16:01:41 Waiting for src bucket item count to get to: 200000 curr count: 161552
2021/12/15 16:01:46 Waiting for src bucket item count to get to: 200000 curr count: 176210
2021/12/15 16:01:51 Waiting for src bucket item count to get to: 200000 curr count: 176210
2021/12/15 16:01:56 Waiting for src bucket item count to get to: 200000 curr count: 191936
2021/12/15 16:02:01 Waiting for src bucket item count to get to: 200000 curr count: 191936
2021/12/15 16:02:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:02:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:02:06 Starting bucket ops to source bucket
2021/12/15 16:05:57 Waiting for dst bucket item count to get to: 0 curr count: 4029
2021/12/15 16:06:02 Waiting for dst bucket item count to get to: 0 curr count: 4029
2021/12/15 16:06:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:06:02 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:06:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:06:07 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 16:06:12 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 16:06:17 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 16:06:22 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/15 16:06:22 App: TestEventingRebInWithOps got un-deployed
2021/12/15 16:06:22 No eventing-con process running
2021/12/15 16:06:22 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/15 16:06:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:06:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:06:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:06:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:06:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.81s)
=== RUN   TestEventingRebOutWithOps
2021/12/15 16:06:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 16:06:41 Removing node(s): [] from the cluster
2021/12/15 16:06:46 Rebalance progress: 100
2021/12/15 16:06:51 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 16:06:51 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 16:06:51 No eventing-con process running
2021/12/15 16:06:51 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/15 16:06:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:06:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:06:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:06:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:07:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:07:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:07:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:07:09 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:07:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 16:07:15 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 16:07:20 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 16:07:25 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 16:07:30 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/15 16:07:30 App: TestEventingRebOutWithOps got deployed
2021/12/15 16:07:30 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:07:30 Starting bucket ops to source bucket
2021/12/15 16:07:35 Rebalance progress: 12.5
2021/12/15 16:07:40 Rebalance progress: 12.5
2021/12/15 16:07:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:07:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:07:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:07:46 Waiting for src bucket item count to get to: 200000 curr count: 13203
2021/12/15 16:07:51 Waiting for src bucket item count to get to: 200000 curr count: 29179
2021/12/15 16:07:56 Waiting for src bucket item count to get to: 200000 curr count: 29179
2021/12/15 16:08:01 Waiting for src bucket item count to get to: 200000 curr count: 45388
2021/12/15 16:08:06 Waiting for src bucket item count to get to: 200000 curr count: 45388
2021/12/15 16:08:11 Waiting for src bucket item count to get to: 200000 curr count: 61543
2021/12/15 16:08:16 Waiting for src bucket item count to get to: 200000 curr count: 61543
2021/12/15 16:08:21 Waiting for src bucket item count to get to: 200000 curr count: 61543
2021/12/15 16:08:26 Waiting for src bucket item count to get to: 200000 curr count: 76640
2021/12/15 16:08:31 Waiting for src bucket item count to get to: 200000 curr count: 76640
2021/12/15 16:08:36 Waiting for src bucket item count to get to: 200000 curr count: 92651
2021/12/15 16:08:41 Waiting for src bucket item count to get to: 200000 curr count: 92651
2021/12/15 16:08:46 Waiting for src bucket item count to get to: 200000 curr count: 108873
2021/12/15 16:08:51 Waiting for src bucket item count to get to: 200000 curr count: 108873
2021/12/15 16:08:56 Waiting for src bucket item count to get to: 200000 curr count: 124879
2021/12/15 16:09:01 Waiting for src bucket item count to get to: 200000 curr count: 124879
2021/12/15 16:09:06 Waiting for src bucket item count to get to: 200000 curr count: 140701
2021/12/15 16:09:11 Waiting for src bucket item count to get to: 200000 curr count: 140701
2021/12/15 16:09:16 Waiting for src bucket item count to get to: 200000 curr count: 156857
2021/12/15 16:09:21 Waiting for src bucket item count to get to: 200000 curr count: 156857
2021/12/15 16:09:26 Waiting for src bucket item count to get to: 200000 curr count: 172227
2021/12/15 16:09:31 Waiting for src bucket item count to get to: 200000 curr count: 172227
2021/12/15 16:09:36 Waiting for src bucket item count to get to: 200000 curr count: 188250
2021/12/15 16:09:41 Waiting for src bucket item count to get to: 200000 curr count: 188250
2021/12/15 16:09:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:09:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:09:46 Starting bucket ops to source bucket
2021/12/15 16:13:36 Waiting for dst bucket item count to get to: 0 curr count: 2721
2021/12/15 16:13:41 Waiting for dst bucket item count to get to: 0 curr count: 2721
2021/12/15 16:13:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:13:41 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:13:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:13:46 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 16:13:51 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 16:13:56 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 16:14:01 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/15 16:14:01 App: TestEventingRebOutWithOps got un-deployed
2021/12/15 16:14:01 No eventing-con process running
2021/12/15 16:14:01 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/15 16:14:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:14:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:14:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:14:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:14:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:14:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (459.27s)
=== RUN   TestEventingSwapRebWithOps
2021/12/15 16:14:25 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 16:14:25 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 16:14:25 No eventing-con process running
2021/12/15 16:14:25 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/15 16:14:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:14:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:14:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:14:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:14:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:14:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:14:43 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:14:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 16:14:49 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 16:14:54 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 16:14:59 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 16:15:04 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/15 16:15:04 App: TestEventingSwapRebWithOps got deployed
2021/12/15 16:15:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 16:15:04 Starting bucket ops to source bucket
2021/12/15 16:15:05 Removing node(s): [] from the cluster
2021/12/15 16:15:10 Rebalance progress: 12.5
2021/12/15 16:15:15 Rebalance progress: 12.5
2021/12/15 16:15:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:15:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 16:15:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:15:25 Rebalance progress: 12.5
2021/12/15 16:15:30 Rebalance progress: 12.5
2021/12/15 16:15:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:15:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 16:15:41 Rebalance progress: 12.5
2021/12/15 16:15:46 Rebalance progress: 12.5
2021/12/15 16:15:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:15:51 Waiting for src bucket item count to get to: 200000 curr count: 52267
2021/12/15 16:15:56 Waiting for src bucket item count to get to: 200000 curr count: 68478
2021/12/15 16:16:01 Waiting for src bucket item count to get to: 200000 curr count: 83200
2021/12/15 16:16:06 Waiting for src bucket item count to get to: 200000 curr count: 83200
2021/12/15 16:16:11 Waiting for src bucket item count to get to: 200000 curr count: 98851
2021/12/15 16:16:16 Waiting for src bucket item count to get to: 200000 curr count: 98851
2021/12/15 16:16:21 Waiting for src bucket item count to get to: 200000 curr count: 115144
2021/12/15 16:16:26 Waiting for src bucket item count to get to: 200000 curr count: 115144
2021/12/15 16:16:31 Waiting for src bucket item count to get to: 200000 curr count: 131359
2021/12/15 16:16:36 Waiting for src bucket item count to get to: 200000 curr count: 131359
2021/12/15 16:16:41 Waiting for src bucket item count to get to: 200000 curr count: 147678
2021/12/15 16:16:46 Waiting for src bucket item count to get to: 200000 curr count: 147678
2021/12/15 16:16:51 Waiting for src bucket item count to get to: 200000 curr count: 163899
2021/12/15 16:16:56 Waiting for src bucket item count to get to: 200000 curr count: 163899
2021/12/15 16:17:01 Waiting for src bucket item count to get to: 200000 curr count: 179087
2021/12/15 16:17:06 Waiting for src bucket item count to get to: 200000 curr count: 179087
2021/12/15 16:17:11 Waiting for src bucket item count to get to: 200000 curr count: 179087
2021/12/15 16:17:16 Waiting for src bucket item count to get to: 200000 curr count: 195073
2021/12/15 16:17:21 Waiting for src bucket item count to get to: 200000 curr count: 195073
2021/12/15 16:17:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:17:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:17:26 Starting bucket ops to source bucket
2021/12/15 16:21:16 Waiting for dst bucket item count to get to: 0 curr count: 21345
2021/12/15 16:21:21 Waiting for dst bucket item count to get to: 0 curr count: 2938
2021/12/15 16:21:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:21:21 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:21:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:21:26 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 16:21:31 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 16:21:36 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 16:21:41 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/15 16:21:41 App: TestEventingSwapRebWithOps got un-deployed
2021/12/15 16:21:41 No eventing-con process running
2021/12/15 16:21:41 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/15 16:21:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:21:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:21:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:21:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:21:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (460.13s)
=== RUN   TestKVRebInWithOps
2021/12/15 16:22:05 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 16:22:05 App: TestKVRebInWithOps got un-deployed
2021/12/15 16:22:05 No eventing-con process running
2021/12/15 16:22:05 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/15 16:22:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:22:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:22:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:22:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:22:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:22:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:22:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:22:23 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:22:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 16:22:29 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 16:22:34 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 16:22:39 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 16:22:44 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/15 16:22:45 App: TestKVRebInWithOps got deployed
2021/12/15 16:22:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 16:22:45 Starting bucket ops to source bucket
2021/12/15 16:22:45 Removing node(s): [] from the cluster
2021/12/15 16:22:50 Rebalance progress: 12.5
2021/12/15 16:22:55 Rebalance progress: 12.5
2021/12/15 16:23:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:23:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:23:05 Rebalance progress: 12.5
2021/12/15 16:23:10 Rebalance progress: 12.5
2021/12/15 16:23:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:23:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:23:15 Waiting for src bucket item count to get to: 200000 curr count: 37406
2021/12/15 16:23:20 Waiting for src bucket item count to get to: 200000 curr count: 37406
2021/12/15 16:23:25 Waiting for src bucket item count to get to: 200000 curr count: 53508
2021/12/15 16:23:30 Waiting for src bucket item count to get to: 200000 curr count: 69676
2021/12/15 16:23:35 Waiting for src bucket item count to get to: 200000 curr count: 69676
2021/12/15 16:23:40 Waiting for src bucket item count to get to: 200000 curr count: 84714
2021/12/15 16:23:45 Waiting for src bucket item count to get to: 200000 curr count: 84714
2021/12/15 16:23:50 Waiting for src bucket item count to get to: 200000 curr count: 84714
2021/12/15 16:23:55 Waiting for src bucket item count to get to: 200000 curr count: 100965
2021/12/15 16:24:00 Waiting for src bucket item count to get to: 200000 curr count: 100965
2021/12/15 16:24:05 Waiting for src bucket item count to get to: 200000 curr count: 117362
2021/12/15 16:24:10 Waiting for src bucket item count to get to: 200000 curr count: 117362
2021/12/15 16:24:15 Waiting for src bucket item count to get to: 200000 curr count: 133301
2021/12/15 16:24:20 Waiting for src bucket item count to get to: 200000 curr count: 133301
2021/12/15 16:24:25 Waiting for src bucket item count to get to: 200000 curr count: 149246
2021/12/15 16:24:30 Waiting for src bucket item count to get to: 200000 curr count: 149246
2021/12/15 16:24:35 Waiting for src bucket item count to get to: 200000 curr count: 164932
2021/12/15 16:24:40 Waiting for src bucket item count to get to: 200000 curr count: 164932
2021/12/15 16:24:45 Waiting for src bucket item count to get to: 200000 curr count: 179464
2021/12/15 16:24:50 Waiting for src bucket item count to get to: 200000 curr count: 179464
2021/12/15 16:24:55 Waiting for src bucket item count to get to: 200000 curr count: 195309
2021/12/15 16:25:00 Waiting for src bucket item count to get to: 200000 curr count: 195309
2021/12/15 16:25:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:25:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:25:05 Starting bucket ops to source bucket
2021/12/15 16:28:55 Waiting for dst bucket item count to get to: 0 curr count: 1410
2021/12/15 16:29:00 Waiting for dst bucket item count to get to: 0 curr count: 1410
2021/12/15 16:29:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:29:00 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:29:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:29:05 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 16:29:10 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 16:29:15 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 16:29:20 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/15 16:29:20 App: TestKVRebInWithOps got un-deployed
2021/12/15 16:29:20 No eventing-con process running
2021/12/15 16:29:20 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/15 16:29:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:29:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:29:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:29:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:29:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.12s)
=== RUN   TestKVRebOutWithOps
2021/12/15 16:29:39 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 16:29:39 Removing node(s): [] from the cluster
2021/12/15 16:29:44 Rebalance progress: 100
2021/12/15 16:29:49 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 16:29:49 App: TestKVRebOutWithOps got un-deployed
2021/12/15 16:29:49 No eventing-con process running
2021/12/15 16:29:49 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/15 16:29:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:29:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:29:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:30:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:30:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:30:08 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:30:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 16:30:14 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 16:30:19 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 16:30:24 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 16:30:29 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/15 16:30:29 App: TestKVRebOutWithOps got deployed
2021/12/15 16:30:29 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:30:29 Starting bucket ops to source bucket
2021/12/15 16:30:34 Rebalance progress: 12.5
2021/12/15 16:30:39 Rebalance progress: 12.5
2021/12/15 16:30:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:30:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:30:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:30:44 Waiting for src bucket item count to get to: 200000 curr count: 15914
2021/12/15 16:30:49 Waiting for src bucket item count to get to: 200000 curr count: 15914
2021/12/15 16:30:54 Waiting for src bucket item count to get to: 200000 curr count: 31760
2021/12/15 16:30:59 Waiting for src bucket item count to get to: 200000 curr count: 31760
2021/12/15 16:31:04 Waiting for src bucket item count to get to: 200000 curr count: 47759
2021/12/15 16:31:09 Waiting for src bucket item count to get to: 200000 curr count: 47759
2021/12/15 16:31:14 Waiting for src bucket item count to get to: 200000 curr count: 47759
2021/12/15 16:31:19 Waiting for src bucket item count to get to: 200000 curr count: 64000
2021/12/15 16:31:24 Waiting for src bucket item count to get to: 200000 curr count: 64000
2021/12/15 16:31:29 Waiting for src bucket item count to get to: 200000 curr count: 78813
2021/12/15 16:31:34 Waiting for src bucket item count to get to: 200000 curr count: 78813
2021/12/15 16:31:39 Waiting for src bucket item count to get to: 200000 curr count: 94463
2021/12/15 16:31:44 Waiting for src bucket item count to get to: 200000 curr count: 94463
2021/12/15 16:31:49 Waiting for src bucket item count to get to: 200000 curr count: 110493
2021/12/15 16:31:54 Waiting for src bucket item count to get to: 200000 curr count: 110493
2021/12/15 16:31:59 Waiting for src bucket item count to get to: 200000 curr count: 126213
2021/12/15 16:32:04 Waiting for src bucket item count to get to: 200000 curr count: 142057
2021/12/15 16:32:09 Waiting for src bucket item count to get to: 200000 curr count: 142057
2021/12/15 16:32:14 Waiting for src bucket item count to get to: 200000 curr count: 158257
2021/12/15 16:32:19 Waiting for src bucket item count to get to: 200000 curr count: 158257
2021/12/15 16:32:24 Waiting for src bucket item count to get to: 200000 curr count: 173579
2021/12/15 16:32:29 Waiting for src bucket item count to get to: 200000 curr count: 173579
2021/12/15 16:32:34 Waiting for src bucket item count to get to: 200000 curr count: 189491
2021/12/15 16:32:39 Waiting for src bucket item count to get to: 200000 curr count: 189491
2021/12/15 16:32:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:32:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:32:44 Starting bucket ops to source bucket
2021/12/15 16:36:34 Waiting for dst bucket item count to get to: 0 curr count: 17934
2021/12/15 16:36:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:36:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:36:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:36:39 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 16:36:44 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 16:36:49 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 16:36:54 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/15 16:36:54 App: TestKVRebOutWithOps got un-deployed
2021/12/15 16:36:54 No eventing-con process running
2021/12/15 16:36:54 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/15 16:36:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:37:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:37:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:37:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:37:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:37:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (454.19s)
=== RUN   TestKVSwapRebWithOps
2021/12/15 16:37:18 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 16:37:18 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 16:37:18 No eventing-con process running
2021/12/15 16:37:18 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/15 16:37:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:37:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:37:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:37:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:37:37 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:37:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/15 16:37:43 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 16:37:48 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 16:37:53 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 16:37:58 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/15 16:37:58 App: TestKVSwapRebWithOps got deployed
2021/12/15 16:37:58 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 16:37:58 Starting bucket ops to source bucket
2021/12/15 16:37:58 Removing node(s): [] from the cluster
2021/12/15 16:38:03 Rebalance progress: 12.5
2021/12/15 16:38:08 Rebalance progress: 12.5
2021/12/15 16:38:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:38:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 16:38:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:38:19 Rebalance progress: 12.5
2021/12/15 16:38:24 Rebalance progress: 12.5
2021/12/15 16:38:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:38:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 16:38:34 Rebalance progress: 12.5
2021/12/15 16:38:39 Rebalance progress: 12.5
2021/12/15 16:38:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:38:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:38:44 Waiting for src bucket item count to get to: 200000 curr count: 63518
2021/12/15 16:38:49 Waiting for src bucket item count to get to: 200000 curr count: 63518
2021/12/15 16:38:54 Waiting for src bucket item count to get to: 200000 curr count: 78359
2021/12/15 16:38:59 Waiting for src bucket item count to get to: 200000 curr count: 78359
2021/12/15 16:39:04 Waiting for src bucket item count to get to: 200000 curr count: 94330
2021/12/15 16:39:09 Waiting for src bucket item count to get to: 200000 curr count: 94330
2021/12/15 16:39:14 Waiting for src bucket item count to get to: 200000 curr count: 110475
2021/12/15 16:39:19 Waiting for src bucket item count to get to: 200000 curr count: 110475
2021/12/15 16:39:24 Waiting for src bucket item count to get to: 200000 curr count: 126730
2021/12/15 16:39:29 Waiting for src bucket item count to get to: 200000 curr count: 126730
2021/12/15 16:39:34 Waiting for src bucket item count to get to: 200000 curr count: 143023
2021/12/15 16:39:39 Waiting for src bucket item count to get to: 200000 curr count: 143023
2021/12/15 16:39:44 Waiting for src bucket item count to get to: 200000 curr count: 159287
2021/12/15 16:39:49 Waiting for src bucket item count to get to: 200000 curr count: 159287
2021/12/15 16:39:54 Waiting for src bucket item count to get to: 200000 curr count: 174514
2021/12/15 16:39:59 Waiting for src bucket item count to get to: 200000 curr count: 174514
2021/12/15 16:40:04 Waiting for src bucket item count to get to: 200000 curr count: 190275
2021/12/15 16:40:09 Waiting for src bucket item count to get to: 200000 curr count: 190275
2021/12/15 16:40:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/15 16:40:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/15 16:40:14 Starting bucket ops to source bucket
2021/12/15 16:44:05 Waiting for dst bucket item count to get to: 0 curr count: 19182
2021/12/15 16:44:10 Waiting for dst bucket item count to get to: 0 curr count: 991
2021/12/15 16:44:15 Waiting for dst bucket item count to get to: 0 curr count: 991
2021/12/15 16:44:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:44:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:44:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:44:20 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 16:44:25 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 16:44:30 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 16:44:35 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/15 16:44:35 App: TestKVSwapRebWithOps got un-deployed
2021/12/15 16:44:35 No eventing-con process running
2021/12/15 16:44:35 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/15 16:44:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:44:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:44:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:44:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:44:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (460.56s)
=== RUN   TestEventingRebInWithTimers
2021/12/15 16:44:59 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 16:44:59 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 16:44:59 No eventing-con process running
2021/12/15 16:44:59 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/15 16:45:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:45:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:45:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:45:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:45:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:45:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:45:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:45:17 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:45:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 16:45:23 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 16:45:28 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 16:45:33 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 16:45:38 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/15 16:45:38 App: TestEventingRebInWithTimers got deployed
2021/12/15 16:45:38 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 16:45:38 Starting bucket ops to source bucket
2021/12/15 16:45:39 Removing node(s): [] from the cluster
2021/12/15 16:45:44 Rebalance progress: 12.5
2021/12/15 16:45:49 Rebalance progress: 12.5
2021/12/15 16:45:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:45:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:45:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:45:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:45:59 Rebalance progress: 12.5
2021/12/15 16:46:04 Rebalance progress: 12.5
2021/12/15 16:46:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:46:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:46:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:46:09 Waiting for src bucket item count to get to: 50000 curr count: 26290
2021/12/15 16:46:14 Waiting for src bucket item count to get to: 50000 curr count: 36284
2021/12/15 16:46:19 Waiting for src bucket item count to get to: 50000 curr count: 36284
2021/12/15 16:46:24 Waiting for src bucket item count to get to: 50000 curr count: 48581
2021/12/15 16:46:29 Waiting for src bucket item count to get to: 50000 curr count: 48581
2021/12/15 16:46:39 Waiting for dst bucket item count to get to: 50000 curr count: 47421
2021/12/15 16:46:44 Waiting for dst bucket item count to get to: 50000 curr count: 47421
2021/12/15 16:46:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 16:46:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 16:46:44 Starting bucket ops to source bucket
2021/12/15 16:47:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:47:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:48:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:48:09 Waiting for dst bucket item count to get to: 0 curr count: 42816
2021/12/15 16:48:14 Waiting for dst bucket item count to get to: 0 curr count: 42816
2021/12/15 16:48:19 Waiting for dst bucket item count to get to: 0 curr count: 26364
2021/12/15 16:48:24 Waiting for dst bucket item count to get to: 0 curr count: 26364
2021/12/15 16:48:29 Waiting for dst bucket item count to get to: 0 curr count: 11955
2021/12/15 16:48:34 Waiting for dst bucket item count to get to: 0 curr count: 11955
2021/12/15 16:48:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:48:34 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:48:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:48:39 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 16:48:44 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 16:48:49 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 16:48:54 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/15 16:48:54 App: TestEventingRebInWithTimers got un-deployed
2021/12/15 16:48:54 No eventing-con process running
2021/12/15 16:48:54 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/15 16:48:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:49:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:49:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:49:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:49:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.34s)
=== RUN   TestEventingRebOutWithTimers
2021/12/15 16:49:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 16:49:13 Removing node(s): [] from the cluster
2021/12/15 16:49:19 Rebalance progress: 100
2021/12/15 16:49:24 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 16:49:24 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 16:49:24 No eventing-con process running
2021/12/15 16:49:24 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/15 16:49:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:49:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:49:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:49:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:49:42 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:49:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 16:49:48 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 16:49:53 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 16:49:58 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 16:50:03 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/15 16:50:03 App: TestEventingRebOutWithTimers got deployed
2021/12/15 16:50:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:50:03 Starting bucket ops to source bucket
2021/12/15 16:50:08 Rebalance progress: 12.5
2021/12/15 16:50:13 Rebalance progress: 12.5
2021/12/15 16:50:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:50:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:50:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:50:18 Waiting for src bucket item count to get to: 50000 curr count: 8031
2021/12/15 16:50:23 Waiting for src bucket item count to get to: 50000 curr count: 20628
2021/12/15 16:50:28 Waiting for src bucket item count to get to: 50000 curr count: 20628
2021/12/15 16:50:33 Waiting for src bucket item count to get to: 50000 curr count: 31711
2021/12/15 16:50:38 Waiting for src bucket item count to get to: 50000 curr count: 31711
2021/12/15 16:50:43 Waiting for src bucket item count to get to: 50000 curr count: 31711
2021/12/15 16:50:48 Waiting for src bucket item count to get to: 50000 curr count: 43084
2021/12/15 16:50:53 Waiting for src bucket item count to get to: 50000 curr count: 43084
2021/12/15 16:51:03 Waiting for dst bucket item count to get to: 50000 curr count: 41749
2021/12/15 16:51:08 Waiting for dst bucket item count to get to: 50000 curr count: 41749
2021/12/15 16:51:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 16:51:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 16:51:08 Starting bucket ops to source bucket
2021/12/15 16:52:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:52:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:52:29 Waiting for dst bucket item count to get to: 0 curr count: 45182
2021/12/15 16:52:34 Waiting for dst bucket item count to get to: 0 curr count: 45182
2021/12/15 16:52:39 Waiting for dst bucket item count to get to: 0 curr count: 37628
2021/12/15 16:52:44 Waiting for dst bucket item count to get to: 0 curr count: 37628
2021/12/15 16:52:49 Waiting for dst bucket item count to get to: 0 curr count: 12651
2021/12/15 16:52:54 Waiting for dst bucket item count to get to: 0 curr count: 12651
2021/12/15 16:52:59 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/15 16:53:04 Waiting for dst bucket item count to get to: 0 curr count: 95
2021/12/15 16:53:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:53:04 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:53:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:53:09 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 16:53:14 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 16:53:19 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 16:53:24 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/15 16:53:24 App: TestEventingRebOutWithTimers got un-deployed
2021/12/15 16:53:24 No eventing-con process running
2021/12/15 16:53:24 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/15 16:53:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:53:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:53:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:53:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:53:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.46s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/15 16:53:47 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 16:53:47 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 16:53:47 No eventing-con process running
2021/12/15 16:53:47 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/15 16:53:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:53:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:53:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:53:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:54:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:54:06 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:54:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 16:54:12 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 16:54:17 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 16:54:22 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 16:54:27 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/15 16:54:27 App: TestEventingSwapRebWithTimers got deployed
2021/12/15 16:54:27 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 16:54:27 Starting bucket ops to source bucket
2021/12/15 16:54:28 Removing node(s): [] from the cluster
2021/12/15 16:54:33 Rebalance progress: 12.5
2021/12/15 16:54:38 Rebalance progress: 12.5
2021/12/15 16:54:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:54:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 16:54:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:54:48 Rebalance progress: 12.5
2021/12/15 16:54:53 Rebalance progress: 12.5
2021/12/15 16:54:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:54:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:54:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 16:55:03 Rebalance progress: 12.5
2021/12/15 16:55:08 Rebalance progress: 12.5
2021/12/15 16:55:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:55:13 Waiting for src bucket item count to get to: 50000 curr count: 49685
2021/12/15 16:55:18 Waiting for src bucket item count to get to: 50000 curr count: 49685
2021/12/15 16:55:28 Waiting for dst bucket item count to get to: 50000 curr count: 41227
2021/12/15 16:55:33 Waiting for dst bucket item count to get to: 50000 curr count: 41227
2021/12/15 16:55:38 Waiting for dst bucket item count to get to: 50000 curr count: 49362
2021/12/15 16:55:43 Waiting for dst bucket item count to get to: 50000 curr count: 49362
2021/12/15 16:55:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 16:55:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 16:55:43 Starting bucket ops to source bucket
2021/12/15 16:56:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:56:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:57:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:57:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 16:57:14 Waiting for dst bucket item count to get to: 0 curr count: 47237
2021/12/15 16:57:19 Waiting for dst bucket item count to get to: 0 curr count: 47237
2021/12/15 16:57:24 Waiting for dst bucket item count to get to: 0 curr count: 29355
2021/12/15 16:57:29 Waiting for dst bucket item count to get to: 0 curr count: 29355
2021/12/15 16:57:34 Waiting for dst bucket item count to get to: 0 curr count: 15519
2021/12/15 16:57:39 Waiting for dst bucket item count to get to: 0 curr count: 15519
2021/12/15 16:57:44 Waiting for dst bucket item count to get to: 0 curr count: 214
2021/12/15 16:57:49 Waiting for dst bucket item count to get to: 0 curr count: 214
2021/12/15 16:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:57:49 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 16:57:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 16:57:54 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 16:57:59 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 16:58:04 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 16:58:09 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/15 16:58:09 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/15 16:58:09 No eventing-con process running
2021/12/15 16:58:09 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/15 16:58:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:58:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:58:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:58:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:58:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (285.08s)
=== RUN   TestKVRebInWithTimers
2021/12/15 16:58:32 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 16:58:32 App: TestKVRebInWithTimers got un-deployed
2021/12/15 16:58:33 No eventing-con process running
2021/12/15 16:58:33 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/15 16:58:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:58:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:58:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 16:58:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:58:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:58:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:58:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 16:58:51 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 16:58:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 16:58:57 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 16:59:02 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 16:59:07 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 16:59:12 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/15 16:59:12 App: TestKVRebInWithTimers got deployed
2021/12/15 16:59:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 16:59:12 Starting bucket ops to source bucket
2021/12/15 16:59:13 Removing node(s): [] from the cluster
2021/12/15 16:59:18 Rebalance progress: 12.5
2021/12/15 16:59:23 Rebalance progress: 12.5
2021/12/15 16:59:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:59:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:59:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:59:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 16:59:33 Rebalance progress: 12.5
2021/12/15 16:59:38 Rebalance progress: 12.5
2021/12/15 16:59:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:59:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 16:59:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 16:59:43 Waiting for src bucket item count to get to: 50000 curr count: 33667
2021/12/15 16:59:48 Waiting for src bucket item count to get to: 50000 curr count: 33667
2021/12/15 16:59:53 Waiting for src bucket item count to get to: 50000 curr count: 43787
2021/12/15 16:59:58 Waiting for src bucket item count to get to: 50000 curr count: 43787
2021/12/15 17:00:08 Waiting for dst bucket item count to get to: 50000 curr count: 37226
2021/12/15 17:00:13 Waiting for dst bucket item count to get to: 50000 curr count: 37226
2021/12/15 17:00:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:00:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:00:13 Starting bucket ops to source bucket
2021/12/15 17:01:23 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:01:28 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:01:33 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:01:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:01:43 Waiting for dst bucket item count to get to: 0 curr count: 44226
2021/12/15 17:01:48 Waiting for dst bucket item count to get to: 0 curr count: 44226
2021/12/15 17:01:53 Waiting for dst bucket item count to get to: 0 curr count: 30352
2021/12/15 17:01:58 Waiting for dst bucket item count to get to: 0 curr count: 30352
2021/12/15 17:02:03 Waiting for dst bucket item count to get to: 0 curr count: 10045
2021/12/15 17:02:08 Waiting for dst bucket item count to get to: 0 curr count: 10045
2021/12/15 17:02:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:02:08 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 17:02:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:02:13 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 17:02:19 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 17:02:24 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 17:02:29 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/15 17:02:29 App: TestKVRebInWithTimers got un-deployed
2021/12/15 17:02:29 No eventing-con process running
2021/12/15 17:02:29 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/15 17:02:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:02:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:02:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:02:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:02:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:02:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (259.71s)
=== RUN   TestKVRebOutWithTimers
2021/12/15 17:02:47 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:02:47 Removing node(s): [] from the cluster
2021/12/15 17:02:52 Rebalance progress: 100
2021/12/15 17:02:57 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 17:02:57 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 17:02:57 No eventing-con process running
2021/12/15 17:02:57 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/15 17:02:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:03:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:03:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:03:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:03:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:03:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:03:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:03:16 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:03:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 17:03:22 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 17:03:27 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 17:03:32 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 17:03:37 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/15 17:03:37 App: TestKVRebOutWithTimers got deployed
2021/12/15 17:03:37 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:03:37 Starting bucket ops to source bucket
2021/12/15 17:03:42 Rebalance progress: 12.5
2021/12/15 17:03:47 Rebalance progress: 12.5
2021/12/15 17:03:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:03:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:03:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:03:52 Waiting for src bucket item count to get to: 50000 curr count: 17009
2021/12/15 17:03:57 Waiting for src bucket item count to get to: 50000 curr count: 17009
2021/12/15 17:04:02 Waiting for src bucket item count to get to: 50000 curr count: 28733
2021/12/15 17:04:07 Waiting for src bucket item count to get to: 50000 curr count: 28733
2021/12/15 17:04:12 Waiting for src bucket item count to get to: 50000 curr count: 38849
2021/12/15 17:04:17 Waiting for src bucket item count to get to: 50000 curr count: 38849
2021/12/15 17:04:22 Waiting for src bucket item count to get to: 50000 curr count: 38849
2021/12/15 17:04:32 Waiting for dst bucket item count to get to: 50000 curr count: 34415
2021/12/15 17:04:37 Waiting for dst bucket item count to get to: 50000 curr count: 34415
2021/12/15 17:04:42 Waiting for dst bucket item count to get to: 50000 curr count: 47137
2021/12/15 17:04:47 Waiting for dst bucket item count to get to: 50000 curr count: 47137
2021/12/15 17:04:47 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:04:47 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:04:47 Starting bucket ops to source bucket
2021/12/15 17:05:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:06:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:06:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:06:13 Waiting for dst bucket item count to get to: 0 curr count: 47375
2021/12/15 17:06:18 Waiting for dst bucket item count to get to: 0 curr count: 47375
2021/12/15 17:06:23 Waiting for dst bucket item count to get to: 0 curr count: 38073
2021/12/15 17:06:28 Waiting for dst bucket item count to get to: 0 curr count: 38073
2021/12/15 17:06:33 Waiting for dst bucket item count to get to: 0 curr count: 16567
2021/12/15 17:06:38 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/15 17:06:43 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/15 17:06:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:06:43 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 17:06:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:06:48 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 17:06:53 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 17:06:58 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 17:07:03 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/15 17:07:03 App: TestKVRebOutWithTimers got un-deployed
2021/12/15 17:07:03 No eventing-con process running
2021/12/15 17:07:03 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/15 17:07:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:07:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:07:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:07:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:07:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:07:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (274.13s)
=== RUN   TestKVSwapRebWithTimers
2021/12/15 17:07:26 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:07:26 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 17:07:26 No eventing-con process running
2021/12/15 17:07:26 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/15 17:07:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:07:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:07:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:07:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:07:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:07:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:07:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:07:45 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:07:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/15 17:07:51 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 17:07:56 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 17:08:01 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 17:08:06 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/15 17:08:06 App: TestKVSwapRebWithTimers got deployed
2021/12/15 17:08:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:08:06 Starting bucket ops to source bucket
2021/12/15 17:08:07 Removing node(s): [] from the cluster
2021/12/15 17:08:12 Rebalance progress: 12.5
2021/12/15 17:08:17 Rebalance progress: 12.5
2021/12/15 17:08:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:08:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 17:08:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:08:28 Rebalance progress: 12.5
2021/12/15 17:08:33 Rebalance progress: 12.5
2021/12/15 17:08:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:08:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:08:43 Rebalance progress: 12.5
2021/12/15 17:08:48 Rebalance progress: 12.5
2021/12/15 17:08:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:08:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:08:53 Waiting for src bucket item count to get to: 50000 curr count: 49508
2021/12/15 17:08:58 Waiting for src bucket item count to get to: 50000 curr count: 49508
2021/12/15 17:09:08 Waiting for dst bucket item count to get to: 50000 curr count: 40862
2021/12/15 17:09:13 Waiting for dst bucket item count to get to: 50000 curr count: 40862
2021/12/15 17:09:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:09:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:09:13 Starting bucket ops to source bucket
2021/12/15 17:10:24 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:10:29 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:10:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:10:39 Waiting for dst bucket item count to get to: 0 curr count: 42975
2021/12/15 17:10:44 Waiting for dst bucket item count to get to: 0 curr count: 42975
2021/12/15 17:10:49 Waiting for dst bucket item count to get to: 0 curr count: 33359
2021/12/15 17:10:54 Waiting for dst bucket item count to get to: 0 curr count: 33359
2021/12/15 17:10:59 Waiting for dst bucket item count to get to: 0 curr count: 12096
2021/12/15 17:11:04 Waiting for dst bucket item count to get to: 0 curr count: 12096
2021/12/15 17:11:09 Waiting for dst bucket item count to get to: 0 curr count: 364
2021/12/15 17:11:14 Waiting for dst bucket item count to get to: 0 curr count: 364
2021/12/15 17:11:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:11:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/15 17:11:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:11:19 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:24 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:29 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:34 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:39 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:44 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/15 17:11:44 App: TestKVSwapRebWithTimers got un-deployed
2021/12/15 17:11:44 No eventing-con process running
2021/12/15 17:11:44 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/15 17:11:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:11:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:11:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:11:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:11:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:11:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (281.28s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/15 17:12:08 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:12:08 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 17:12:08 No eventing-con process running
2021/12/15 17:12:08 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/15 17:12:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:12:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:12:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:12:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:12:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:12:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:12:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:12:26 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:12:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 17:12:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 17:12:37 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 17:12:42 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 17:12:47 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/15 17:12:47 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/15 17:12:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:12:47 Starting bucket ops to source bucket
2021/12/15 17:12:48 Removing node(s): [] from the cluster
2021/12/15 17:12:53 Rebalance progress: 12.5
2021/12/15 17:12:58 Rebalance progress: 12.5
2021/12/15 17:13:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:13:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:13:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:13:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:13:08 Rebalance progress: 12.5
2021/12/15 17:13:13 Rebalance progress: 12.5
2021/12/15 17:13:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:13:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:13:18 Waiting for src bucket item count to get to: 50000 curr count: 26797
2021/12/15 17:13:23 Waiting for src bucket item count to get to: 50000 curr count: 36949
2021/12/15 17:13:28 Waiting for src bucket item count to get to: 50000 curr count: 36949
2021/12/15 17:13:33 Waiting for src bucket item count to get to: 50000 curr count: 47782
2021/12/15 17:13:38 Waiting for src bucket item count to get to: 50000 curr count: 47782
2021/12/15 17:13:48 Waiting for dst bucket item count to get to: 50000 curr count: 39359
2021/12/15 17:13:53 Waiting for dst bucket item count to get to: 50000 curr count: 39359
2021/12/15 17:13:58 Waiting for dst bucket item count to get to: 50000 curr count: 39359
2021/12/15 17:14:03 Waiting for dst bucket item count to get to: 50000 curr count: 46532
2021/12/15 17:14:08 Waiting for dst bucket item count to get to: 50000 curr count: 46532
2021/12/15 17:14:08 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:14:08 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:14:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:14:13 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:18 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:23 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/15 17:14:38 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/15 17:14:38 No eventing-con process running
2021/12/15 17:14:38 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/15 17:14:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:14:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:14:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:14:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:14:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (179.08s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/15 17:15:02 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:15:02 Removing node(s): [] from the cluster
2021/12/15 17:15:07 Rebalance progress: 100
2021/12/15 17:15:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:15:12 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 17:15:12 No eventing-con process running
2021/12/15 17:15:12 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/15 17:15:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:15:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:15:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:15:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:15:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:15:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:15:31 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:15:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 17:15:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 17:15:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 17:15:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 17:15:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/15 17:15:52 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/15 17:15:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:15:52 Starting bucket ops to source bucket
2021/12/15 17:15:57 Rebalance progress: 12.5
2021/12/15 17:16:02 Rebalance progress: 12.5
2021/12/15 17:16:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:16:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:16:07 Waiting for src bucket item count to get to: 50000 curr count: 9657
2021/12/15 17:16:12 Waiting for src bucket item count to get to: 50000 curr count: 23608
2021/12/15 17:16:17 Waiting for src bucket item count to get to: 50000 curr count: 23608
2021/12/15 17:16:22 Waiting for src bucket item count to get to: 50000 curr count: 36247
2021/12/15 17:16:27 Waiting for src bucket item count to get to: 50000 curr count: 36247
2021/12/15 17:16:32 Waiting for src bucket item count to get to: 50000 curr count: 45288
2021/12/15 17:16:37 Waiting for src bucket item count to get to: 50000 curr count: 45288
2021/12/15 17:16:42 Waiting for src bucket item count to get to: 50000 curr count: 45288
2021/12/15 17:16:52 Waiting for dst bucket item count to get to: 50000 curr count: 32519
2021/12/15 17:16:57 Waiting for dst bucket item count to get to: 50000 curr count: 32519
2021/12/15 17:17:02 Waiting for dst bucket item count to get to: 50000 curr count: 46466
2021/12/15 17:17:07 Waiting for dst bucket item count to get to: 50000 curr count: 46466
2021/12/15 17:17:07 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:17:07 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:17:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:17:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:17 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:22 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:27 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/15 17:17:37 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/15 17:17:37 No eventing-con process running
2021/12/15 17:17:37 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/15 17:17:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:17:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:17:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:17:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:17:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:18:01 Waiting for dst bucket item count to get to: 0 curr count: 46103
2021/12/15 17:18:06 Waiting for dst bucket item count to get to: 0 curr count: 46103
2021/12/15 17:18:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.24s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/15 17:18:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:18:16 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 17:18:16 No eventing-con process running
2021/12/15 17:18:16 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/15 17:18:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:18:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:18:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:18:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:18:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:18:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:18:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:18:35 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:18:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 17:18:41 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 17:18:46 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 17:18:51 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 17:18:56 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/15 17:18:56 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/15 17:18:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:18:56 Starting bucket ops to source bucket
2021/12/15 17:18:56 Removing node(s): [] from the cluster
2021/12/15 17:19:01 Rebalance progress: 12.5
2021/12/15 17:19:06 Rebalance progress: 12.5
2021/12/15 17:19:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:19:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 17:19:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:19:17 Rebalance progress: 12.5
2021/12/15 17:19:22 Rebalance progress: 12.5
2021/12/15 17:19:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:19:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:19:32 Rebalance progress: 12.5
2021/12/15 17:19:37 Rebalance progress: 12.5
2021/12/15 17:19:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:19:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:19:42 Waiting for src bucket item count to get to: 50000 curr count: 49334
2021/12/15 17:19:47 Waiting for src bucket item count to get to: 50000 curr count: 49334
2021/12/15 17:19:57 Waiting for dst bucket item count to get to: 50000 curr count: 35099
2021/12/15 17:20:02 Waiting for dst bucket item count to get to: 50000 curr count: 35099
2021/12/15 17:20:07 Waiting for dst bucket item count to get to: 50000 curr count: 46192
2021/12/15 17:20:12 Waiting for dst bucket item count to get to: 50000 curr count: 46192
2021/12/15 17:20:12 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:20:12 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:20:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:20:17 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:22 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/15 17:20:42 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/15 17:20:42 No eventing-con process running
2021/12/15 17:20:43 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/15 17:20:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:20:54 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:20:59 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:21:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:21:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:21:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:21:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (190.15s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/15 17:21:26 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:21:26 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 17:21:26 No eventing-con process running
2021/12/15 17:21:26 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/15 17:21:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:21:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:21:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:21:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:21:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:21:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:21:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:21:45 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:21:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 17:21:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 17:21:56 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 17:22:01 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 17:22:06 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/15 17:22:06 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/15 17:22:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:22:06 Starting bucket ops to source bucket
2021/12/15 17:22:06 Removing node(s): [] from the cluster
2021/12/15 17:22:11 Rebalance progress: 12.5
2021/12/15 17:22:16 Rebalance progress: 12.5
2021/12/15 17:22:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:22:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:22:26 Rebalance progress: 12.5
2021/12/15 17:22:31 Rebalance progress: 12.5
2021/12/15 17:22:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:22:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:22:36 Waiting for src bucket item count to get to: 50000 curr count: 30846
2021/12/15 17:22:41 Waiting for src bucket item count to get to: 50000 curr count: 41881
2021/12/15 17:22:46 Waiting for src bucket item count to get to: 50000 curr count: 41881
2021/12/15 17:22:56 Waiting for dst bucket item count to get to: 50000 curr count: 28231
2021/12/15 17:23:01 Waiting for dst bucket item count to get to: 50000 curr count: 28231
2021/12/15 17:23:06 Waiting for dst bucket item count to get to: 50000 curr count: 36039
2021/12/15 17:23:11 Waiting for dst bucket item count to get to: 50000 curr count: 36039
2021/12/15 17:23:16 Waiting for dst bucket item count to get to: 50000 curr count: 47773
2021/12/15 17:23:21 Waiting for dst bucket item count to get to: 50000 curr count: 47773
2021/12/15 17:23:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:23:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:23:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:23:27 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:32 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:37 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:42 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:47 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:52 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/15 17:23:52 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/15 17:23:52 No eventing-con process running
2021/12/15 17:23:52 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/15 17:23:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:24:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:24:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:24:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:24:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:24:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:24:20 Waiting for dst bucket item count to get to: 0 curr count: 23495
2021/12/15 17:24:25 Waiting for dst bucket item count to get to: 0 curr count: 23495
2021/12/15 17:24:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (189.33s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/15 17:24:30 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:24:31 Removing node(s): [] from the cluster
2021/12/15 17:24:36 Rebalance progress: 100
2021/12/15 17:24:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:24:41 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 17:24:41 No eventing-con process running
2021/12/15 17:24:41 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/15 17:24:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:24:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:24:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:24:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:24:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:24:59 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:25:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 17:25:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 17:25:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 17:25:15 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 17:25:20 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/15 17:25:20 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/15 17:25:20 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:25:20 Starting bucket ops to source bucket
2021/12/15 17:25:25 Rebalance progress: 12.5
2021/12/15 17:25:30 Rebalance progress: 12.5
2021/12/15 17:25:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:25:35 Waiting for src bucket item count to get to: 50000 curr count: 11264
2021/12/15 17:25:40 Waiting for src bucket item count to get to: 50000 curr count: 24699
2021/12/15 17:25:46 Waiting for src bucket item count to get to: 50000 curr count: 24699
2021/12/15 17:25:51 Waiting for src bucket item count to get to: 50000 curr count: 36681
2021/12/15 17:25:56 Waiting for src bucket item count to get to: 50000 curr count: 36681
2021/12/15 17:26:01 Waiting for src bucket item count to get to: 50000 curr count: 46465
2021/12/15 17:26:06 Waiting for src bucket item count to get to: 50000 curr count: 46465
2021/12/15 17:26:11 Waiting for src bucket item count to get to: 50000 curr count: 46465
2021/12/15 17:26:21 Waiting for dst bucket item count to get to: 50000 curr count: 32433
2021/12/15 17:26:26 Waiting for dst bucket item count to get to: 50000 curr count: 32433
2021/12/15 17:26:31 Waiting for dst bucket item count to get to: 50000 curr count: 42598
2021/12/15 17:26:36 Waiting for dst bucket item count to get to: 50000 curr count: 42598
2021/12/15 17:26:36 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:26:36 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:26:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:26:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:26:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:26:51 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:26:56 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:27:01 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:27:06 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/15 17:27:06 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/15 17:27:06 No eventing-con process running
2021/12/15 17:27:06 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/15 17:27:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:27:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:27:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:27:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:27:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:27:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:27:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.20s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/15 17:27:35 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:27:35 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 17:27:35 No eventing-con process running
2021/12/15 17:27:35 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/15 17:27:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:27:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:27:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:27:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:27:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:27:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:27:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:27:53 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:27:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 17:27:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 17:28:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 17:28:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 17:28:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/15 17:28:14 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/15 17:28:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:28:14 Starting bucket ops to source bucket
2021/12/15 17:28:15 Removing node(s): [] from the cluster
2021/12/15 17:28:20 Rebalance progress: 12.5
2021/12/15 17:28:25 Rebalance progress: 12.5
2021/12/15 17:28:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:28:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:28:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:28:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 17:28:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:28:35 Rebalance progress: 12.5
2021/12/15 17:28:40 Rebalance progress: 12.5
2021/12/15 17:28:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:28:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:28:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:28:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:28:50 Rebalance progress: 12.5
2021/12/15 17:28:55 Rebalance progress: 12.5
2021/12/15 17:29:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:29:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:29:00 Waiting for src bucket item count to get to: 50000 curr count: 41264
2021/12/15 17:29:10 Waiting for dst bucket item count to get to: 50000 curr count: 30717
2021/12/15 17:29:15 Waiting for dst bucket item count to get to: 50000 curr count: 36624
2021/12/15 17:29:20 Waiting for dst bucket item count to get to: 50000 curr count: 36624
2021/12/15 17:29:20 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:29:20 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:29:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:29:26 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:31 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:36 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:41 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:46 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:51 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/15 17:29:51 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/15 17:29:51 No eventing-con process running
2021/12/15 17:29:51 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/15 17:29:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:29:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:30:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:30:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:30:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:30:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:30:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:30:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (169.93s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/15 17:30:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:30:25 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 17:30:25 No eventing-con process running
2021/12/15 17:30:25 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/15 17:30:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:30:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:30:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:30:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:30:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:30:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:30:43 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:30:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 17:30:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 17:30:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 17:30:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 17:31:04 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/15 17:31:04 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/15 17:31:04 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:31:04 Starting bucket ops to source bucket
2021/12/15 17:31:05 Removing node(s): [] from the cluster
2021/12/15 17:31:10 Rebalance progress: 12.5
2021/12/15 17:31:15 Rebalance progress: 12.5
2021/12/15 17:31:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:31:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:31:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:31:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:31:25 Rebalance progress: 12.5
2021/12/15 17:31:30 Rebalance progress: 12.5
2021/12/15 17:31:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:31:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:31:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:31:35 Waiting for src bucket item count to get to: 50000 curr count: 30071
2021/12/15 17:31:40 Waiting for src bucket item count to get to: 50000 curr count: 30071
2021/12/15 17:31:45 Waiting for src bucket item count to get to: 50000 curr count: 41183
2021/12/15 17:31:50 Waiting for src bucket item count to get to: 50000 curr count: 41183
2021/12/15 17:32:00 Waiting for dst bucket item count to get to: 50000 curr count: 33381
2021/12/15 17:32:05 Waiting for dst bucket item count to get to: 50000 curr count: 43231
2021/12/15 17:32:10 Waiting for dst bucket item count to get to: 50000 curr count: 43231
2021/12/15 17:32:10 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:32:10 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:32:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:32:15 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:30 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:36 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:41 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/15 17:32:41 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/15 17:32:41 No eventing-con process running
2021/12/15 17:32:41 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/15 17:32:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:32:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:32:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:32:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:32:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:32:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:33:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:33:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:33:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (174.63s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/15 17:33:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:33:14 Removing node(s): [] from the cluster
2021/12/15 17:33:19 Rebalance progress: 100
2021/12/15 17:33:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:33:24 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 17:33:25 No eventing-con process running
2021/12/15 17:33:25 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/15 17:33:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:33:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:33:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:33:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:33:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:33:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:33:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:33:43 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:33:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 17:33:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 17:33:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 17:33:59 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 17:34:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/15 17:34:04 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/15 17:34:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:34:04 Starting bucket ops to source bucket
2021/12/15 17:34:09 Rebalance progress: 12.5
2021/12/15 17:34:14 Rebalance progress: 12.5
2021/12/15 17:34:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:34:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:34:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:34:19 Waiting for src bucket item count to get to: 50000 curr count: 6255
2021/12/15 17:34:24 Waiting for src bucket item count to get to: 50000 curr count: 19348
2021/12/15 17:34:29 Waiting for src bucket item count to get to: 50000 curr count: 19348
2021/12/15 17:34:34 Waiting for src bucket item count to get to: 50000 curr count: 30245
2021/12/15 17:34:39 Waiting for src bucket item count to get to: 50000 curr count: 30245
2021/12/15 17:34:44 Waiting for src bucket item count to get to: 50000 curr count: 30245
2021/12/15 17:34:49 Waiting for src bucket item count to get to: 50000 curr count: 41761
2021/12/15 17:34:54 Waiting for src bucket item count to get to: 50000 curr count: 41761
2021/12/15 17:35:04 Waiting for dst bucket item count to get to: 50000 curr count: 38213
2021/12/15 17:35:09 Waiting for dst bucket item count to get to: 50000 curr count: 46489
2021/12/15 17:35:14 Waiting for dst bucket item count to get to: 50000 curr count: 46489
2021/12/15 17:35:14 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:35:14 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:35:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:35:20 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:25 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:30 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:35 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:40 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:45 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/15 17:35:45 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/15 17:35:45 No eventing-con process running
2021/12/15 17:35:45 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/15 17:35:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:35:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:35:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:35:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:35:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:36:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:36:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:36:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.29s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/15 17:36:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:36:18 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 17:36:18 No eventing-con process running
2021/12/15 17:36:18 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/15 17:36:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:36:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:36:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:36:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:36:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:36:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:36:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:36:37 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:36:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 17:36:43 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 17:36:48 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 17:36:53 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 17:36:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/15 17:36:58 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/15 17:36:58 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/15 17:36:58 Starting bucket ops to source bucket
2021/12/15 17:36:58 Removing node(s): [] from the cluster
2021/12/15 17:37:04 Rebalance progress: 12.5
2021/12/15 17:37:09 Rebalance progress: 12.5
2021/12/15 17:37:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:37:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 17:37:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:37:19 Rebalance progress: 12.5
2021/12/15 17:37:24 Rebalance progress: 12.5
2021/12/15 17:37:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:37:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:37:34 Rebalance progress: 12.5
2021/12/15 17:37:39 Rebalance progress: 12.5
2021/12/15 17:37:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:37:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:37:44 Waiting for src bucket item count to get to: 50000 curr count: 47205
2021/12/15 17:37:49 Waiting for src bucket item count to get to: 50000 curr count: 47205
2021/12/15 17:37:59 Waiting for dst bucket item count to get to: 50000 curr count: 46013
2021/12/15 17:38:04 Waiting for dst bucket item count to get to: 50000 curr count: 46013
2021/12/15 17:38:04 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:38:04 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:38:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:38:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:19 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:25 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:30 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:35 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/15 17:38:35 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/15 17:38:35 No eventing-con process running
2021/12/15 17:38:35 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/15 17:38:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:38:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:38:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:38:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:38:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (169.79s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/15 17:39:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:39:08 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 17:39:08 No eventing-con process running
2021/12/15 17:39:08 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/15 17:39:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:39:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:39:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:39:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:39:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:39:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:39:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:39:27 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:39:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/15 17:39:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 17:39:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 17:39:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 17:39:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/15 17:39:48 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/15 17:39:48 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:39:48 Starting bucket ops to source bucket
2021/12/15 17:39:48 Removing node(s): [] from the cluster
2021/12/15 17:39:53 Rebalance progress: 12.5
2021/12/15 17:39:58 Rebalance progress: 12.5
2021/12/15 17:40:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:40:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:40:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:40:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:40:09 Rebalance progress: 12.5
2021/12/15 17:40:14 Rebalance progress: 12.5
2021/12/15 17:40:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:40:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:40:19 Waiting for src bucket item count to get to: 50000 curr count: 26690
2021/12/15 17:40:24 Waiting for src bucket item count to get to: 50000 curr count: 36504
2021/12/15 17:40:29 Waiting for src bucket item count to get to: 50000 curr count: 36504
2021/12/15 17:40:34 Waiting for src bucket item count to get to: 50000 curr count: 48629
2021/12/15 17:40:39 Waiting for src bucket item count to get to: 50000 curr count: 48629
2021/12/15 17:40:49 Waiting for dst bucket item count to get to: 50000 curr count: 44607
2021/12/15 17:40:54 Waiting for dst bucket item count to get to: 50000 curr count: 44607
2021/12/15 17:40:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:40:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:40:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:40:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:04 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:09 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:14 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:19 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:24 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/15 17:41:24 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/15 17:41:24 No eventing-con process running
2021/12/15 17:41:24 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/15 17:41:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:41:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:41:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:41:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:41:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:41:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:41:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:41:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.33s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/15 17:41:53 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:41:53 Removing node(s): [] from the cluster
2021/12/15 17:41:58 Rebalance progress: 100
2021/12/15 17:42:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:42:03 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 17:42:03 No eventing-con process running
2021/12/15 17:42:03 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/15 17:42:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:42:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:42:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:42:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:42:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:42:22 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:42:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/15 17:42:28 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 17:42:33 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 17:42:38 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 17:42:43 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/15 17:42:43 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/15 17:42:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:42:43 Starting bucket ops to source bucket
2021/12/15 17:42:48 Rebalance progress: 12.5
2021/12/15 17:42:53 Rebalance progress: 12.5
2021/12/15 17:42:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:42:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:42:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:42:58 Waiting for src bucket item count to get to: 50000 curr count: 8631
2021/12/15 17:43:03 Waiting for src bucket item count to get to: 50000 curr count: 21830
2021/12/15 17:43:08 Waiting for src bucket item count to get to: 50000 curr count: 21830
2021/12/15 17:43:13 Waiting for src bucket item count to get to: 50000 curr count: 33524
2021/12/15 17:43:18 Waiting for src bucket item count to get to: 50000 curr count: 33524
2021/12/15 17:43:23 Waiting for src bucket item count to get to: 50000 curr count: 33524
2021/12/15 17:43:28 Waiting for src bucket item count to get to: 50000 curr count: 44057
2021/12/15 17:43:33 Waiting for src bucket item count to get to: 50000 curr count: 44057
2021/12/15 17:43:43 Waiting for dst bucket item count to get to: 50000 curr count: 38392
2021/12/15 17:43:48 Waiting for dst bucket item count to get to: 50000 curr count: 38392
2021/12/15 17:43:48 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:43:48 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:43:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:43:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:43:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:44:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:44:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:44:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:44:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/15 17:44:18 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/15 17:44:18 No eventing-con process running
2021/12/15 17:44:18 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/15 17:44:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:44:30 Waiting for dst bucket item count to get to: 0 curr count: 15165
2021/12/15 17:44:35 Waiting for dst bucket item count to get to: 0 curr count: 15165
2021/12/15 17:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:44:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:44:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:44:47 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:44:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.48s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/15 17:44:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:44:57 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 17:44:57 No eventing-con process running
2021/12/15 17:44:57 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/15 17:44:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:45:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:45:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:45:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:45:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:45:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:45:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:45:16 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:45:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/15 17:45:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 17:45:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 17:45:32 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 17:45:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/15 17:45:37 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/15 17:45:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:45:37 Starting bucket ops to source bucket
2021/12/15 17:45:37 Removing node(s): [] from the cluster
2021/12/15 17:45:42 Rebalance progress: 12.5
2021/12/15 17:45:47 Rebalance progress: 12.5
2021/12/15 17:45:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:45:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:45:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 17:45:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:45:58 Rebalance progress: 12.5
2021/12/15 17:46:03 Rebalance progress: 12.5
2021/12/15 17:46:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:46:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:46:13 Rebalance progress: 12.5
2021/12/15 17:46:18 Rebalance progress: 12.5
2021/12/15 17:46:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:46:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:46:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:46:23 Waiting for src bucket item count to get to: 50000 curr count: 47207
2021/12/15 17:46:28 Waiting for src bucket item count to get to: 50000 curr count: 47207
2021/12/15 17:46:38 Waiting for dst bucket item count to get to: 50000 curr count: 44229
2021/12/15 17:46:43 Waiting for dst bucket item count to get to: 50000 curr count: 44229
2021/12/15 17:46:43 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/15 17:46:43 Post SET ops. Expected item count: 50000 got 50000
2021/12/15 17:46:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:46:48 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:46:53 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:46:58 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:47:03 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:47:08 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:47:13 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/15 17:47:13 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/15 17:47:13 No eventing-con process running
2021/12/15 17:47:13 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/15 17:47:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:47:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:47:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:47:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:47:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:47:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/15 17:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (169.97s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/15 17:47:47 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:47:47 App: bucket_op_on_update got un-deployed
2021/12/15 17:47:47 No eventing-con process running
2021/12/15 17:47:47 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 17:47:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:47:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:47:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:48:06 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:48:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 17:48:12 Starting bucket ops to source bucket
2021/12/15 17:48:17 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:48:22 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:48:27 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:48:27 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:48:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:48:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:48:32 Rebalance progress: 12.5
2021/12/15 17:48:37 Rebalance progress: 12.5
2021/12/15 17:48:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:48:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 17:48:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:48:48 Rebalance progress: 12.5
2021/12/15 17:48:53 Rebalance progress: 12.5
2021/12/15 17:48:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:48:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:48:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 17:48:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:49:03 Rebalance progress: 12.5
2021/12/15 17:49:08 Rebalance progress: 12.5
2021/12/15 17:49:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:49:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:49:18 Rebalance progress: 12.5
2021/12/15 17:49:23 Rebalance progress: 12.5
2021/12/15 17:49:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:49:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 17:49:33 Rebalance progress: 12.5
2021/12/15 17:49:38 Rebalance progress: 12.5
2021/12/15 17:49:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:49:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 17:49:49 Rebalance progress: 12.5
2021/12/15 17:49:54 Rebalance progress: 12.5
2021/12/15 17:49:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:49:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:49:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:50:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:50:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:50:29 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:50:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 17:50:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:50:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:50:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7379
2021/12/15 17:50:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7379
2021/12/15 17:50:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:50:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:50:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:50:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10668
2021/12/15 17:50:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.42s)
=== RUN   TestKVForceFailover
2021/12/15 17:51:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:51:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:51:12 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:51:12 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 17:51:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:51:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:51:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:51:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:51:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:51:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:51:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:51:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:51:31 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:51:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 17:51:32 Starting bucket ops to source bucket
2021/12/15 17:51:37 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:51:42 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:51:48 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:51:53 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:51:53 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:51:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:51:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:51:53 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 17:52:13 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 17:52:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/15 17:52:18 Rebalance progress: 12.5
2021/12/15 17:52:23 Rebalance progress: 12.5
2021/12/15 17:52:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:52:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:52:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/15 17:52:33 Rebalance progress: 12.5
2021/12/15 17:52:38 Rebalance progress: 12.5
2021/12/15 17:52:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:52:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:52:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:52:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:52:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:53:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:53:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:53:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:53:13 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:53:13 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:53:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 17:53:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:53:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:53:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7056
2021/12/15 17:53:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:53:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:53:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:53:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7056
2021/12/15 17:53:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.60s)
=== RUN   TestKVHardFailoverRecovery
2021/12/15 17:53:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:53:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:53:47 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:53:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 17:53:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:53:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:53:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:53:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:54:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:54:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:54:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 17:54:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 17:54:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:54:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:54:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:54:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:54:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:54:27 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:54:32 Rebalance progress: 12.5
2021/12/15 17:54:37 Rebalance progress: 12.5
2021/12/15 17:54:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:54:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:54:42 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/15 17:55:42 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/15 17:55:47 Rebalance progress: 12.5
2021/12/15 17:55:52 Rebalance progress: 12.5
2021/12/15 17:55:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:55:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:55:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 17:56:02 Rebalance progress: 12.5
2021/12/15 17:56:07 Rebalance progress: 12.5
2021/12/15 17:56:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:56:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:56:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:56:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:56:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:56:43 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:56:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 17:56:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:56:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:56:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12057
2021/12/15 17:56:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:56:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:57:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12057
2021/12/15 17:57:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.44s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/15 17:57:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:57:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:57:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:57:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 17:57:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:57:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:57:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:57:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:57:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:57:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 17:57:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 17:57:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:57:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:57:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:57:56 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:57:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:57:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 17:57:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 17:57:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 17:57:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 17:58:02 Rebalance progress: 12.5
2021/12/15 17:58:07 Rebalance progress: 12.5
2021/12/15 17:58:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:58:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/15 17:58:18 Rebalance progress: 12.5
2021/12/15 17:58:23 Rebalance progress: 12.5
2021/12/15 17:58:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 17:58:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 17:58:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:58:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:58:58 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:58:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 17:59:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:59:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:59:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:59:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:59:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:59:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4616
2021/12/15 17:59:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (129.95s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/15 17:59:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 17:59:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 17:59:26 eventingSetup.go:46: No eventing-con process running
2021/12/15 17:59:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 17:59:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:59:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:59:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 17:59:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:59:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 17:59:40 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 17:59:40 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 17:59:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 17:59:46 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 17:59:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 17:59:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:00:01 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:00:01 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:00:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:00:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:00:07 Rebalance progress: 12.5
2021/12/15 18:00:12 Rebalance progress: 12.5
2021/12/15 18:00:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:00:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 18:00:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 18:00:22 Rebalance progress: 12.5
2021/12/15 18:00:27 Rebalance progress: 12.5
2021/12/15 18:00:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:00:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 18:00:37 Rebalance progress: 12.5
2021/12/15 18:00:42 Rebalance progress: 12.5
2021/12/15 18:00:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:00:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:00:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:00:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:01:18 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:01:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:01:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:01:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:01:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:01:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:01:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:01:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6104
2021/12/15 18:01:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:01:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:01:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:01:41 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:01:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 18:01:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:01:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:01:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:01:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:01:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:01:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (158.54s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/15 18:02:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:02:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:02:05 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:02:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 18:02:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:02:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:02:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:02:24 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 18:02:24 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:02:24 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 18:02:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:02:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:37 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:02:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:02:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:02:38 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:02:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 18:02:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 18:02:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:02:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:02:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:02:59 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 18:03:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:03:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:03:14 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:03:15 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:03:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:03:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:03:20 Rebalance progress: 12.5
2021/12/15 18:03:25 Rebalance progress: 12.5
2021/12/15 18:03:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:03:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 18:03:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:03:35 Rebalance progress: 12.5
2021/12/15 18:03:40 Rebalance progress: 12.5
2021/12/15 18:03:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:03:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:03:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/15 18:03:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:03:51 Rebalance progress: 12.5
2021/12/15 18:03:56 Rebalance progress: 12.5
2021/12/15 18:04:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:04:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 18:04:06 Rebalance progress: 12.5
2021/12/15 18:04:11 Rebalance progress: 12.5
2021/12/15 18:04:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:04:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 18:04:22 Rebalance progress: 12.5
2021/12/15 18:04:27 Rebalance progress: 12.5
2021/12/15 18:04:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:04:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/15 18:04:37 Rebalance progress: 12.5
2021/12/15 18:04:42 Rebalance progress: 12.5
2021/12/15 18:04:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:04:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:04:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:04:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:04:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:05:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:05:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:05:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:05:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:05:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:05:17 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:05:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:05:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:05:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:05:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2303
2021/12/15 18:05:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2303
2021/12/15 18:05:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:05:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:05:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:05:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12218
2021/12/15 18:05:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:05:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:05:54 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:05:59 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:06:04 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:06:09 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:06:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:06:19 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:06:19 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 18:06:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:06:19 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 18:06:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:06:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:06:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:06:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:06:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:06:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (277.54s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/15 18:06:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:06:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:06:43 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:06:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 18:06:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:06:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:06:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:06:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:06:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:06:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:07:01 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:07:01 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 18:07:01 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:07:01 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/15 18:07:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:07:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:07:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:07:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:07:15 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:07:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:07:16 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:07:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/15 18:07:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 18:07:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:07:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:07:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:07:37 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/15 18:07:42 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:07:47 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:07:52 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/15 18:07:52 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:07:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:07:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:07:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:07:58 Rebalance progress: 12.5
2021/12/15 18:08:03 Rebalance progress: 12.5
2021/12/15 18:08:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:08:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 18:08:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 18:08:13 Rebalance progress: 12.5
2021/12/15 18:08:18 Rebalance progress: 12.5
2021/12/15 18:08:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:08:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 18:08:28 Rebalance progress: 12.5
2021/12/15 18:08:33 Rebalance progress: 12.5
2021/12/15 18:08:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:08:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:08:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:08:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:08:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:08:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:08:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:09:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:09:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:09:09 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:09:09 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:09:09 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:09:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:09:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:09:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6027
2021/12/15 18:09:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6027
2021/12/15 18:09:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:09:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:09:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:09:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:09:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:09:40 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:09:45 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:09:50 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:09:55 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:10:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/15 18:10:00 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/15 18:10:00 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:10:00 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/15 18:10:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:10:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:10:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:10:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:10:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:10:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (221.27s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/15 18:10:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:10:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:10:24 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:10:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 18:10:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:10:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:10:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:10:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:10:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:10:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:10:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:10:42 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:10:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 18:10:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 18:10:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:10:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:11:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:11:04 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:11:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:11:04 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:11:24 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:11:34 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:11:39 Rebalance progress: 12.5
2021/12/15 18:11:44 Rebalance progress: 12.5
2021/12/15 18:11:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:11:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:11:49 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 18:11:50 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 18:11:55 Rebalance progress: 12.5
2021/12/15 18:12:00 Rebalance progress: 12.5
2021/12/15 18:12:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:12:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:12:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 18:12:10 Rebalance progress: 12.5
2021/12/15 18:12:15 Rebalance progress: 12.5
2021/12/15 18:12:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:12:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:12:20 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:12:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:12:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:12:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:12:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:12:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:12:45 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:12:45 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:12:45 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:12:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:12:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:12:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9128
2021/12/15 18:12:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:12:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:13:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9128
2021/12/15 18:13:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.18s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/15 18:13:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:13:19 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:13:19 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:13:19 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/15 18:13:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:13:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:13:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:13:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:13:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:13:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 18:13:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/15 18:13:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:13:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:13:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:13:59 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:13:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:13:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:13:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:14:04 Rebalance progress: 12.5
2021/12/15 18:14:09 Rebalance progress: 12.5
2021/12/15 18:14:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:14:14 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:14:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:14:29 Rebalance progress: 12.5
2021/12/15 18:14:34 Rebalance progress: 12.5
2021/12/15 18:14:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:14:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/15 18:14:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/15 18:14:45 Rebalance progress: 12.5
2021/12/15 18:14:50 Rebalance progress: 12.5
2021/12/15 18:14:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:14:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:14:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/15 18:15:00 Rebalance progress: 12.5
2021/12/15 18:15:05 Rebalance progress: 12.5
2021/12/15 18:15:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:15:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:15:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:15:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:15:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:15:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:15:40 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:15:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:15:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:15:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:15:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8588
2021/12/15 18:15:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8588
2021/12/15 18:15:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:15:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:16:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:16:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8588
2021/12/15 18:16:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.79s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/15 18:16:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/15 18:16:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/15 18:16:19 Rebalance progress: 100
2021/12/15 18:16:19 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:16:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:16:49 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/15 18:16:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/15 18:16:55 Starting bucket ops to source bucket
2021/12/15 18:16:55 Removing node(s): [] from the cluster
2021/12/15 18:17:00 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/15 18:17:05 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:17:10 Waiting for app: bucket_op_on_update to get deployed
2021/12/15 18:17:10 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/15 18:17:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/15 18:17:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/15 18:17:41 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/15 18:17:41 eventingSetup.go:46: No eventing-con process running
2021/12/15 18:17:41 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/15 18:17:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:17:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:17:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1498
2021/12/15 18:17:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1498
2021/12/15 18:17:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/15 18:18:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/15 18:18:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/15 18:18:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (115.84s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12100.594s

Rebalance tests took 3 hours, 21 minutes and 43 seconds.

Testing Succeeded


Version: versions-15.12.2021-10.28.cfg
Build Log: make-15.12.2021-10.28.log
Server Log: logs-15.12.2021-10.28.tar.gz

Finished