Building

Started building at 2021/12/16 02:25:37
Calculating base
Updating mirror
Basing run on 7.1.0-1921 5f4063a07
Updating tree for run 16.12.2021-02.25
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at f0d7a1e, changes since last good build: 
 f0d7a1ee Merge remote-tracking branch 'couchbase/unstable' into HEAD
 510c15ed MB-50006 Force snapshot generation with maintFlushTs from timekeeper
kv_engine is at 0811a49, changes since last good build: 
 0811a494e MB-49727: Rename deleteExpiredItem() funcs into processExpiredItem()
 854b47046 MB-48777: Prevent race between shutting down warmup and changing state
 f1e7f3a7e MB-49858: Compaction throttle: account for #writer threads
 fe84d94f4 MB-46378: Use atomic version of NonNegativeCounter only where needed
ns_server is at adbe234, changes since last good build: 
 adbe234d5 MB-48369 Handle status '0' error from /postUILogin
 a07a9442a MB-49969: check isGroupsAvailable on injected poolDefault
couchstore is at 6eea212, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at c1e9d9b, changes since last good build: none
Switching eventing to unstable
eventing is at 589c223, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 8058bdf1] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 37d7dccd] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 7f8e3bf2] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 906621a7] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/6
Total 16 (delta 12), reused 16 (delta 12)
[unstable 77860b96] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/11
Total 49 (delta 40), reused 49 (delta 40)
[unstable 0d7080a6] MB-46304: Add functionality to disable curl feature
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 21 files changed, 193 insertions(+), 31 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/7
Total 59 (delta 49), reused 59 (delta 49)
[unstable 6f5a8092] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167542 at refs/changes/42/167542/5
Total 74 (delta 62), reused 74 (delta 62)
[unstable 723f70d5] MB-49570: Start dcp stream for a single collection id
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:27:25 2021 +0530
 7 files changed, 52 insertions(+), 29 deletions(-)
Pulling in uncommitted change 167544 at refs/changes/44/167544/4
Total 82 (delta 68), reused 82 (delta 68)
[unstable 98051c68] MB-49570: Add backlog stats on ui
 Author: AnkitPrabhu 
 Date: Tue Dec 14 14:28:01 2021 +0530
 2 files changed, 18 insertions(+), 1 deletion(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/4
Total 5 (delta 2), reused 5 (delta 2)
[unstable 057fe0e] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/16 02:43:53
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/16 02:45:24 Started functional
2021/12/16 02:45:33 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 02:45:46 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/16 02:45:46 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 02:45:52 Removing node(s): [] from the cluster
2021/12/16 02:45:57 Rebalance progress: 2.314814814814815
2021/12/16 02:46:02 Rebalance progress: 6.944444444444444
2021/12/16 02:46:07 Rebalance progress: 10.19965277777778
2021/12/16 02:46:12 Rebalance progress: 14.86545138888889
2021/12/16 02:46:17 Rebalance progress: 18.51851851851852
2021/12/16 02:46:22 Rebalance progress: 22.78645833333333
2021/12/16 02:46:27 Rebalance progress: 27.48842592592592
2021/12/16 02:46:32 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/16 02:46:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:46:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:47:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:47:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:47:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:47:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:48:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:48:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:48:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:48:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:49:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:49:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:49:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:49:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/16 02:50:07 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:50:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:50:14 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 02:50:19 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 02:50:24 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 02:50:29 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/16 02:50:29 App: TestAdvancedGetOps got deployed
2021/12/16 02:50:29 Starting bucket ops to source bucket
2021/12/16 02:50:34 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 02:50:39 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 02:50:44 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 02:50:44 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 02:50:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:50:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:50:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:50:49 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 02:50:54 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 02:50:59 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 02:51:04 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/16 02:51:04 App: TestAdvancedGetOps got un-deployed
2021/12/16 02:51:04 No eventing-con process running
2021/12/16 02:51:04 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/16 02:51:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:51:16 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 02:51:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:51:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:51:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 02:51:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (290.08s)
=== RUN   TestAdvancedInsertOps
2021/12/16 02:51:27 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:51:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:51:33 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:51:38 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:51:43 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:51:48 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:51:49 App: TestAdvancedInsertOps got deployed
2021/12/16 02:51:49 Starting bucket ops to source bucket
2021/12/16 02:51:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 02:51:59 Waiting for dst bucket item count to get to: 1024 curr count: 434
2021/12/16 02:52:04 Waiting for dst bucket item count to get to: 1024 curr count: 946
2021/12/16 02:52:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 02:52:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:52:04 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:52:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:52:10 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:52:15 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:52:20 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:52:25 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:52:25 App: TestAdvancedInsertOps got un-deployed
2021/12/16 02:52:25 No eventing-con process running
2021/12/16 02:52:25 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 02:52:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:52:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 02:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:52:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:52:48 Waiting for dst bucket item count to get to: 0 curr count: 741
2021/12/16 02:52:53 Waiting for dst bucket item count to get to: 0 curr count: 229
2021/12/16 02:52:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:52:53 Testing insert operation on source bucket
2021/12/16 02:52:53 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:52:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:52:59 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:53:04 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:53:09 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:53:14 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/16 02:53:14 App: TestAdvancedInsertOps got deployed
2021/12/16 02:53:14 Starting bucket ops to source bucket
2021/12/16 02:53:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 02:53:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 02:53:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 02:53:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:53:25 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:53:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:53:30 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:53:35 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:53:40 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:53:45 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/16 02:53:45 App: TestAdvancedInsertOps got un-deployed
2021/12/16 02:53:45 No eventing-con process running
2021/12/16 02:53:46 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/16 02:53:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:53:57 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 02:53:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:53:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:54:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.70s)
=== RUN   TestAdvancedUpsertOps
2021/12/16 02:54:04 Starting bucket ops to source bucket
2021/12/16 02:54:05 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:54:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:54:11 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:54:16 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:54:21 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:54:26 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:54:26 App: TestAdvancedUpsertOps got deployed
2021/12/16 02:54:26 Starting bucket ops to source bucket
2021/12/16 02:54:32 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 02:54:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:54:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:54:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:54:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:54:37 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:54:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:54:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:54:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:54:52 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 02:54:52 No eventing-con process running
2021/12/16 02:54:52 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 02:54:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:55:04 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 02:55:09 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 02:55:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:55:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:55:16 Testing upsert operation on source bucket
2021/12/16 02:55:16 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:55:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:55:22 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:55:27 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:55:32 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:55:37 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/16 02:55:37 App: TestAdvancedUpsertOps got deployed
2021/12/16 02:55:37 Starting bucket ops to source bucket
2021/12/16 02:55:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:55:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:55:38 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:55:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:55:43 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:55:48 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:55:53 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:55:58 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/16 02:55:58 App: TestAdvancedUpsertOps got un-deployed
2021/12/16 02:55:58 No eventing-con process running
2021/12/16 02:55:58 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/16 02:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:56:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:56:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (128.03s)
=== RUN   TestAdvancedReplaceOps
2021/12/16 02:56:12 Starting bucket ops to source bucket
2021/12/16 02:56:13 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:56:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:56:19 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:56:24 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:56:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:56:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:56:34 App: TestAdvancedReplaceOps got deployed
2021/12/16 02:56:34 Starting bucket ops to source bucket
2021/12/16 02:56:40 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 02:56:45 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/16 02:56:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:56:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:56:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:56:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:56:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:56:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:57:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:57:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:57:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 02:57:05 No eventing-con process running
2021/12/16 02:57:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 02:57:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:57:17 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 02:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:57:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:57:24 Testing upsert operation on source bucket
2021/12/16 02:57:24 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:57:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:57:30 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:57:35 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:57:40 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:57:45 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/16 02:57:45 App: TestAdvancedReplaceOps got deployed
2021/12/16 02:57:45 Starting bucket ops to source bucket
2021/12/16 02:57:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:57:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:57:46 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:57:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:57:51 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:57:56 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:58:01 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:58:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/16 02:58:07 App: TestAdvancedReplaceOps got un-deployed
2021/12/16 02:58:07 No eventing-con process running
2021/12/16 02:58:07 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/16 02:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:58:18 Waiting for dst bucket item count to get to: 0 curr count: 155
2021/12/16 02:58:23 Waiting for dst bucket item count to get to: 0 curr count: 48
2021/12/16 02:58:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:58:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:58:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (138.01s)
=== RUN   TestAdvancedDeleteOps
2021/12/16 02:58:30 Starting bucket ops to source bucket
2021/12/16 02:58:30 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:58:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:58:36 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 02:58:41 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 02:58:46 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 02:58:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 02:58:51 App: TestAdvancedDeleteOps got deployed
2021/12/16 02:58:51 Starting bucket ops to source bucket
2021/12/16 02:58:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 02:59:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 02:59:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:59:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 02:59:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 02:59:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 02:59:07 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 02:59:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 02:59:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 02:59:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 02:59:22 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 02:59:22 No eventing-con process running
2021/12/16 02:59:22 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 02:59:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:59:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 02:59:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 02:59:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:59:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 02:59:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 02:59:45 Testing delete operation on source bucket
2021/12/16 02:59:45 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 02:59:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/16 02:59:51 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 02:59:56 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 03:00:01 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 03:00:06 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/16 03:00:06 App: TestAdvancedDeleteOps got deployed
2021/12/16 03:00:06 Starting bucket ops to source bucket
2021/12/16 03:00:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:00:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:00:07 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:00:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:00:12 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 03:00:17 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 03:00:22 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 03:00:27 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/16 03:00:27 App: TestAdvancedDeleteOps got un-deployed
2021/12/16 03:00:27 No eventing-con process running
2021/12/16 03:00:27 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/16 03:00:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:00:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:00:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:00:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (130.64s)
=== RUN   TestEnoentAdvancedGet
2021/12/16 03:00:41 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:00:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:00:47 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:00:52 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:00:57 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:01:02 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:01:02 App: TestEnoentAdvancedGet got deployed
2021/12/16 03:01:02 Starting bucket ops to source bucket
2021/12/16 03:01:07 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:01:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:01:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:01:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:01:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:01:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:01:17 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:01:22 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:01:27 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:01:32 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:01:32 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 03:01:32 No eventing-con process running
2021/12/16 03:01:32 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 03:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:01:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:01:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:01:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:02:01 Waiting for dst bucket item count to get to: 0 curr count: 90
2021/12/16 03:02:06 Waiting for dst bucket item count to get to: 0 curr count: 90
2021/12/16 03:02:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:02:06 Testing get enoent operation on source bucket
2021/12/16 03:02:06 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:02:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:02:12 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:02:17 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:02:22 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:02:27 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/16 03:02:27 App: TestEnoentAdvancedGet got deployed
2021/12/16 03:02:27 Starting bucket ops to source bucket
2021/12/16 03:02:32 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 03:02:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 03:02:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:02:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:02:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:02:37 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:02:42 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:02:47 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:02:52 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/16 03:02:52 App: TestEnoentAdvancedGet got un-deployed
2021/12/16 03:02:52 No eventing-con process running
2021/12/16 03:02:52 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/16 03:02:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:03:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 03:03:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:03:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:03:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (154.98s)
=== RUN   TestEnoentAdvancedReplace
2021/12/16 03:03:16 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:03:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:03:22 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:03:27 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:03:32 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:03:37 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:03:37 App: TestEnoentAdvancedReplace got deployed
2021/12/16 03:03:37 Starting bucket ops to source bucket
2021/12/16 03:03:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:03:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:03:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:03:42 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:03:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:03:47 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:03:52 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:03:57 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:04:02 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:04:02 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 03:04:02 No eventing-con process running
2021/12/16 03:04:02 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 03:04:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:04:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:04:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:04:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:04:31 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/16 03:04:36 Waiting for dst bucket item count to get to: 0 curr count: 87
2021/12/16 03:04:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:04:36 Testing get enoent operation on source bucket
2021/12/16 03:04:36 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:04:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:04:42 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:04:47 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:04:52 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:04:57 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/16 03:04:57 App: TestEnoentAdvancedReplace got deployed
2021/12/16 03:04:57 Starting bucket ops to source bucket
2021/12/16 03:05:02 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 03:05:02 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 03:05:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:05:02 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:05:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:05:07 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:05:12 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:05:17 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:05:22 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/16 03:05:22 App: TestEnoentAdvancedReplace got un-deployed
2021/12/16 03:05:22 No eventing-con process running
2021/12/16 03:05:22 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/16 03:05:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:05:34 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 03:05:39 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.00s)
=== RUN   TestEnoentAdvancedDelete
2021/12/16 03:05:46 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:05:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:05:52 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:05:57 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:06:02 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:06:07 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:06:07 App: TestEnoentAdvancedDelete got deployed
2021/12/16 03:06:07 Starting bucket ops to source bucket
2021/12/16 03:06:12 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:06:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:06:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:06:12 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:06:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:06:17 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:06:22 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:06:27 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:06:32 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:06:32 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 03:06:32 No eventing-con process running
2021/12/16 03:06:32 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 03:06:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:06:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:06:49 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:06:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:06:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:07:01 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/16 03:07:06 Waiting for dst bucket item count to get to: 0 curr count: 89
2021/12/16 03:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:07:06 Testing get enoent operation on source bucket
2021/12/16 03:07:06 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:07:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:07:12 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:07:17 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:07:22 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:07:27 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/16 03:07:27 App: TestEnoentAdvancedDelete got deployed
2021/12/16 03:07:27 Starting bucket ops to source bucket
2021/12/16 03:07:32 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/16 03:07:32 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/16 03:07:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:07:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:07:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:07:37 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:07:42 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:07:47 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:07:52 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/16 03:07:52 App: TestEnoentAdvancedDelete got un-deployed
2021/12/16 03:07:52 No eventing-con process running
2021/12/16 03:07:52 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/16 03:07:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:08:04 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 03:08:09 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:08:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:08:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:08:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.08s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/16 03:08:16 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:08:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 03:08:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:08:27 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:08:32 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:08:37 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:08:37 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 03:08:37 Starting bucket ops to source bucket
2021/12/16 03:08:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:08:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:08:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:08:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:08:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:08:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:08:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:08:57 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:08:57 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 03:08:57 No eventing-con process running
2021/12/16 03:08:57 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 03:08:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:09:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:09:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:09:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:09:11 Pumping document in the destination bucket
2021/12/16 03:09:11 Starting bucket ops to source bucket
2021/12/16 03:09:12 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:09:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/16 03:09:18 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:09:23 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:09:28 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:09:33 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/16 03:09:33 App: TestAdvancedInsertKeyExist got deployed
2021/12/16 03:09:33 Starting bucket ops to source bucket
2021/12/16 03:09:38 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:09:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:09:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:09:43 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:09:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:09:48 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:09:53 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:09:58 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:10:03 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/16 03:10:03 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/16 03:10:03 No eventing-con process running
2021/12/16 03:10:03 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/16 03:10:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:10:15 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:10:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (125.71s)
=== RUN   TestExipryGet
2021/12/16 03:10:22 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:10:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:10:28 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:10:33 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:10:38 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:10:43 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:10:43 App: TestExipryGet got deployed
2021/12/16 03:10:43 Starting bucket ops to source bucket
2021/12/16 03:11:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:11:33 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:11:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:11:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:11:33 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:11:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:11:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:11:43 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:11:48 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:11:53 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:11:53 App: TestExipryGet got un-deployed
2021/12/16 03:11:53 No eventing-con process running
2021/12/16 03:11:53 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 03:11:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:12:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:12:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:12:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:12:07 Pumping document in the destination bucket for expiry get
2021/12/16 03:12:07 Starting bucket ops to source bucket
2021/12/16 03:12:07 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:12:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:12:14 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:12:19 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:12:24 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:12:29 Waiting for app: TestExipryGet to get deployed
2021/12/16 03:12:29 App: TestExipryGet got deployed
2021/12/16 03:12:29 Starting bucket ops to source bucket
2021/12/16 03:13:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:13:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:13:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:13:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:13:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:13:24 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:13:29 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:13:34 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:13:39 Waiting for app: TestExipryGet to get un-deployed
2021/12/16 03:13:39 App: TestExipryGet got un-deployed
2021/12/16 03:13:39 No eventing-con process running
2021/12/16 03:13:39 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/16 03:13:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:13:51 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/16 03:13:56 Waiting for dst bucket item count to get to: 0 curr count: 88
2021/12/16 03:13:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:13:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:14:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.95s)
=== RUN   TestExipryInsert
2021/12/16 03:14:02 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:14:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 03:14:09 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:14:14 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:14:19 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:14:24 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:14:24 App: TestExipryInsert got deployed
2021/12/16 03:14:24 Starting bucket ops to source bucket
2021/12/16 03:15:09 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/16 03:15:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:15:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:15:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:15:14 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:15:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:15:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:15:24 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:15:29 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:15:34 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:15:34 App: TestExipryInsert got un-deployed
2021/12/16 03:15:34 No eventing-con process running
2021/12/16 03:15:34 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 03:15:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:15:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:15:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:15:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:15:48 Testing on the destination bucket for expiry insert
2021/12/16 03:15:48 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:15:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/16 03:15:54 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:15:59 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:16:04 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:16:09 Waiting for app: TestExipryInsert to get deployed
2021/12/16 03:16:09 App: TestExipryInsert got deployed
2021/12/16 03:16:09 Starting bucket ops to source bucket
2021/12/16 03:16:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:17:00 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:17:05 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:17:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:17:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:17:05 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:17:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:17:10 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:17:15 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:17:20 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:17:25 Waiting for app: TestExipryInsert to get un-deployed
2021/12/16 03:17:25 App: TestExipryInsert got un-deployed
2021/12/16 03:17:25 No eventing-con process running
2021/12/16 03:17:25 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/16 03:17:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:17:37 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:17:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:17:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:17:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.83s)
=== RUN   TestExipryReplace
2021/12/16 03:17:43 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:17:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:17:49 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:17:54 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:17:59 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:18:04 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:18:04 App: TestExipryReplace got deployed
2021/12/16 03:18:04 Starting bucket ops to source bucket
2021/12/16 03:18:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:18:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:18:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:18:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:18:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:19:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:19:05 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:19:10 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:19:15 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:19:15 App: TestExipryReplace got un-deployed
2021/12/16 03:19:15 No eventing-con process running
2021/12/16 03:19:15 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 03:19:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:19:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:19:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:19:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:19:29 Testing on the destination bucket for expiry insert
2021/12/16 03:19:29 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:19:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:19:35 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:19:40 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:19:45 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:19:50 Waiting for app: TestExipryReplace to get deployed
2021/12/16 03:19:50 App: TestExipryReplace got deployed
2021/12/16 03:19:50 Starting bucket ops to source bucket
2021/12/16 03:20:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:20:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:20:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:20:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:20:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:20:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:20:45 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:20:50 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:20:55 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:21:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/16 03:21:00 App: TestExipryReplace got un-deployed
2021/12/16 03:21:00 No eventing-con process running
2021/12/16 03:21:00 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/16 03:21:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:21:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:21:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:21:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:21:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:21:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.52s)
=== RUN   TestExipryUpsert
2021/12/16 03:21:24 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:21:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 03:21:30 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:21:35 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:21:40 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:21:45 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:21:45 App: TestExipryUpsert got deployed
2021/12/16 03:21:45 Starting bucket ops to source bucket
2021/12/16 03:22:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:22:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:22:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:22:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:22:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:22:35 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:22:40 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:22:45 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:22:50 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:22:50 App: TestExipryUpsert got un-deployed
2021/12/16 03:22:50 No eventing-con process running
2021/12/16 03:22:50 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 03:22:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:22:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:23:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:23:04 Testing on the destination bucket for expiry insert
2021/12/16 03:23:04 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:23:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/16 03:23:11 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:23:16 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:23:21 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:23:26 Waiting for app: TestExipryUpsert to get deployed
2021/12/16 03:23:26 App: TestExipryUpsert got deployed
2021/12/16 03:23:26 Starting bucket ops to source bucket
2021/12/16 03:24:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:24:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:24:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:24:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:24:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:24:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:24:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:24:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/16 03:24:31 App: TestExipryUpsert got un-deployed
2021/12/16 03:24:31 No eventing-con process running
2021/12/16 03:24:31 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/16 03:24:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:24:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:24:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:24:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:24:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:24:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.60s)
=== RUN   TestCasReplace
2021/12/16 03:24:54 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:24:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:25:01 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:25:06 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:25:11 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:25:16 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:25:16 App: TestCasReplace got deployed
2021/12/16 03:25:16 Starting bucket ops to source bucket
2021/12/16 03:25:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:25:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:25:16 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:25:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:25:21 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:25:26 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:25:31 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:25:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:25:36 App: TestCasReplace got un-deployed
2021/12/16 03:25:36 No eventing-con process running
2021/12/16 03:25:36 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 03:25:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:25:48 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/16 03:25:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:25:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:25:55 Testing on the destination bucket for upsert with cas
2021/12/16 03:25:55 Starting bucket ops to source bucket
2021/12/16 03:25:55 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:25:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/16 03:26:01 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:26:06 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:26:11 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:26:16 Waiting for app: TestCasReplace to get deployed
2021/12/16 03:26:16 App: TestCasReplace got deployed
2021/12/16 03:26:16 Starting bucket ops to source bucket
2021/12/16 03:26:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:26:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:26:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:26:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:26:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:26:26 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:26:31 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:26:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:26:41 Waiting for app: TestCasReplace to get un-deployed
2021/12/16 03:26:41 App: TestCasReplace got un-deployed
2021/12/16 03:26:41 No eventing-con process running
2021/12/16 03:26:41 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/16 03:26:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:26:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:26:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:26:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:27:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.43s)
=== RUN   TestCasDelete
2021/12/16 03:27:05 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:27:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:27:11 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:27:16 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:27:21 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:27:26 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:27:26 App: TestCasDelete got deployed
2021/12/16 03:27:26 Starting bucket ops to source bucket
2021/12/16 03:27:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:27:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:27:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:27:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:27:31 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:27:36 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:27:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:27:46 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:27:46 App: TestCasDelete got un-deployed
2021/12/16 03:27:46 No eventing-con process running
2021/12/16 03:27:46 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 03:27:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:27:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:28:00 Testing on the destination bucket for delete with cas
2021/12/16 03:28:00 Starting bucket ops to source bucket
2021/12/16 03:28:01 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:28:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:28:07 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:28:12 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:28:17 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:28:22 Waiting for app: TestCasDelete to get deployed
2021/12/16 03:28:22 App: TestCasDelete got deployed
2021/12/16 03:28:22 Starting bucket ops to source bucket
2021/12/16 03:28:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:28:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:28:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:28:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:28:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 03:28:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:28:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:28:42 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:28:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:28:52 Waiting for app: TestCasDelete to get un-deployed
2021/12/16 03:28:52 App: TestCasDelete got un-deployed
2021/12/16 03:28:52 No eventing-con process running
2021/12/16 03:28:52 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/16 03:28:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:29:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:29:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:29:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:29:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:29:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.89s)
=== RUN   TestCountersIncrement
2021/12/16 03:29:16 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 03:29:22 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 03:29:22 Removing node(s): [] from the cluster
2021/12/16 03:29:27 Rebalance progress: 100
2021/12/16 03:29:27 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:29:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 03:29:33 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:29:38 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:29:43 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:29:48 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:29:48 App: TestCountersIncrement got deployed
2021/12/16 03:29:48 Starting bucket ops to source bucket
2021/12/16 03:29:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 03:29:59 Waiting for dst bucket item count to get to: 1024 curr count: 466
2021/12/16 03:30:04 Waiting for dst bucket item count to get to: 1024 curr count: 466
2021/12/16 03:30:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 03:30:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:30:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:30:10 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:30:15 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:30:20 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:30:25 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:30:25 App: TestCountersIncrement got un-deployed
2021/12/16 03:30:25 No eventing-con process running
2021/12/16 03:30:25 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 03:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:30:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/16 03:30:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:30:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:30:48 Waiting for dst bucket item count to get to: 0 curr count: 765
2021/12/16 03:30:53 Waiting for dst bucket item count to get to: 0 curr count: 253
2021/12/16 03:30:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:30:53 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:30:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/16 03:30:59 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:31:04 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:31:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:31:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/16 03:31:14 App: TestCountersIncrement got deployed
2021/12/16 03:31:14 Starting bucket ops to source bucket
2021/12/16 03:31:20 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 03:31:25 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 03:31:25 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 03:31:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:31:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:31:31 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:31:36 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:31:41 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:31:46 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/16 03:31:46 App: TestCountersIncrement got un-deployed
2021/12/16 03:31:46 No eventing-con process running
2021/12/16 03:31:46 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/16 03:31:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:31:57 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 03:31:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:31:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:32:09 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 03:32:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:32:09 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 03:32:14 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.56s)
=== RUN   TestCountersDecrement
2021/12/16 03:32:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 03:32:15 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 03:32:15 Removing node(s): [] from the cluster
2021/12/16 03:32:20 Rebalance progress: 100
2021/12/16 03:32:20 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:32:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 03:32:26 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:32:31 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:32:36 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:32:41 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:32:41 App: TestCountersDecrement got deployed
2021/12/16 03:32:41 Starting bucket ops to source bucket
2021/12/16 03:32:47 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 03:32:52 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/16 03:32:52 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/16 03:32:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:32:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:32:58 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:33:03 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:33:08 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:33:13 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:33:13 App: TestCountersDecrement got un-deployed
2021/12/16 03:33:13 No eventing-con process running
2021/12/16 03:33:13 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 03:33:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:33:24 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/16 03:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:33:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:33:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/16 03:33:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:33:36 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:33:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/16 03:33:42 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:33:47 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:33:52 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:33:57 Waiting for app: TestCountersDecrement to get deployed
2021/12/16 03:33:57 App: TestCountersDecrement got deployed
2021/12/16 03:33:57 Starting bucket ops to source bucket
2021/12/16 03:34:04 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/16 03:34:09 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 03:34:14 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/16 03:34:14 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/16 03:34:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:34:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:34:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:34:24 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:34:29 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:34:34 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/16 03:34:34 App: TestCountersDecrement got un-deployed
2021/12/16 03:34:34 No eventing-con process running
2021/12/16 03:34:34 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/16 03:34:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:34:46 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/16 03:34:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:34:57 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 03:34:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:34:57 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/16 03:35:02 Rebalance progress: 100
--- PASS: TestCountersDecrement (168.16s)
=== RUN   TestCacheGet
2021/12/16 03:35:02 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:35:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:35:09 Waiting for app: TestCacheGet to get deployed
2021/12/16 03:35:14 Waiting for app: TestCacheGet to get deployed
2021/12/16 03:35:19 Waiting for app: TestCacheGet to get deployed
2021/12/16 03:35:24 Waiting for app: TestCacheGet to get deployed
2021/12/16 03:35:24 App: TestCacheGet got deployed
2021/12/16 03:35:24 Starting bucket ops to source bucket
2021/12/16 03:35:29 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 03:35:34 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 03:35:39 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/16 03:35:44 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/16 03:35:49 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/16 03:35:54 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/16 03:35:54 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/16 03:35:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:35:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:35:59 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 03:36:04 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 03:36:09 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 03:36:14 Waiting for app: TestCacheGet to get un-deployed
2021/12/16 03:36:14 App: TestCacheGet got un-deployed
2021/12/16 03:36:14 No eventing-con process running
2021/12/16 03:36:14 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/16 03:36:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:36:26 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/16 03:36:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:36:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:36:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/16 03:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.24s)
=== RUN   TestCacheRYOW
2021/12/16 03:36:38 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:36:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/16 03:36:44 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 03:36:49 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 03:36:54 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 03:36:59 Waiting for app: TestCacheRYOW to get deployed
2021/12/16 03:36:59 App: TestCacheRYOW got deployed
2021/12/16 03:36:59 Starting bucket ops to source bucket
2021/12/16 03:37:04 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/16 03:37:09 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/16 03:37:09 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/16 03:37:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:37:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:37:14 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 03:37:19 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 03:37:24 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 03:37:29 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/16 03:37:29 App: TestCacheRYOW got un-deployed
2021/12/16 03:37:29 No eventing-con process running
2021/12/16 03:37:29 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/16 03:37:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:37:41 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:37:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:37:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:37:58 Waiting for dst bucket item count to get to: 0 curr count: 29
2021/12/16 03:38:03 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/16 03:38:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (85.14s)
=== RUN   TestCacheExpiry
2021/12/16 03:38:03 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:38:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/16 03:38:09 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 03:38:14 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 03:38:19 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 03:38:24 Waiting for app: TestCacheExpiry to get deployed
2021/12/16 03:38:24 App: TestCacheExpiry got deployed
2021/12/16 03:38:24 Starting bucket ops to source bucket
2021/12/16 03:38:29 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:38:34 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/16 03:38:39 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/16 03:38:44 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/16 03:38:49 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/16 03:38:49 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:38:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:38:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:38:54 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 03:39:00 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 03:39:05 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 03:39:10 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/16 03:39:10 App: TestCacheExpiry got un-deployed
2021/12/16 03:39:10 No eventing-con process running
2021/12/16 03:39:10 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/16 03:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:39:21 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:39:26 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:39:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:39:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:39:38 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/16 03:39:43 Waiting for dst bucket item count to get to: 0 curr count: 24
2021/12/16 03:39:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.37s)
=== RUN   TestCacheExpiryCustom
2021/12/16 03:39:43 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:39:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 03:39:49 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 03:39:54 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 03:39:59 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 03:40:04 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/16 03:40:04 App: TestCacheExpiryCustom got deployed
2021/12/16 03:40:04 Starting bucket ops to source bucket
2021/12/16 03:40:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:40:15 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/16 03:40:20 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/16 03:40:25 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/16 03:40:30 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/16 03:40:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:40:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:40:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:40:35 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 03:40:40 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 03:40:45 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 03:40:50 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/16 03:40:50 App: TestCacheExpiryCustom got un-deployed
2021/12/16 03:40:50 No eventing-con process running
2021/12/16 03:40:50 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/16 03:40:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:41:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:41:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:41:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:41:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:41:18 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/16 03:41:23 Waiting for dst bucket item count to get to: 0 curr count: 30
2021/12/16 03:41:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.15s)
=== RUN   TestCacheOverflowCustom
2021/12/16 03:41:23 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:41:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/16 03:41:30 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 03:41:35 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 03:41:40 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 03:41:45 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/16 03:41:45 App: TestCacheOverflowCustom got deployed
2021/12/16 03:41:45 Starting bucket ops to source bucket
2021/12/16 03:41:50 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/16 03:41:55 Waiting for dst bucket item count to get to: 35 curr count: 7
2021/12/16 03:42:00 Waiting for dst bucket item count to get to: 35 curr count: 16
2021/12/16 03:42:05 Waiting for dst bucket item count to get to: 35 curr count: 23
2021/12/16 03:42:10 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/16 03:42:15 Waiting for dst bucket item count to get to: 35 curr count: 30
2021/12/16 03:42:15 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/16 03:42:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:42:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:42:20 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 03:42:25 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 03:42:30 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 03:42:35 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/16 03:42:35 App: TestCacheOverflowCustom got un-deployed
2021/12/16 03:42:35 No eventing-con process running
2021/12/16 03:42:35 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/16 03:42:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:42:47 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/16 03:42:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:42:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:42:59 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/16 03:43:04 Waiting for dst bucket item count to get to: 0 curr count: 13
2021/12/16 03:43:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.25s)
=== RUN   TestStrictMode
2021/12/16 03:43:04 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:43:05 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.07s)
=== RUN   TestDataTypes
2021/12/16 03:43:05 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:43:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/16 03:43:11 Waiting for app: TestDataTypes to get deployed
2021/12/16 03:43:16 Waiting for app: TestDataTypes to get deployed
2021/12/16 03:43:21 Waiting for app: TestDataTypes to get deployed
2021/12/16 03:43:26 Waiting for app: TestDataTypes to get deployed
2021/12/16 03:43:26 App: TestDataTypes got deployed
2021/12/16 03:43:26 Starting bucket ops to source bucket
2021/12/16 03:43:31 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/16 03:43:36 Waiting for dst bucket item count to get to: 18 curr count: 8
2021/12/16 03:43:36 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/16 03:43:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:43:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:43:41 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 03:43:46 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 03:43:51 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 03:43:56 Waiting for app: TestDataTypes to get un-deployed
2021/12/16 03:43:56 App: TestDataTypes got un-deployed
2021/12/16 03:43:56 No eventing-con process running
2021/12/16 03:43:56 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/16 03:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:44:08 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 03:44:13 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 03:44:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:44:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:44:25 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/16 03:44:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.09s)
=== RUN   TestEnoentGet
2021/12/16 03:44:25 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:44:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:44:31 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:44:36 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:44:41 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:44:46 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:44:46 App: TestEnoentGet got deployed
2021/12/16 03:44:46 Starting bucket ops to source bucket
2021/12/16 03:44:51 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:44:51 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:44:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:44:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:44:56 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:45:01 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:45:06 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:45:11 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:45:11 App: TestEnoentGet got un-deployed
2021/12/16 03:45:11 No eventing-con process running
2021/12/16 03:45:11 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 03:45:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:45:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:45:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:45:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:45:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:45:40 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/16 03:45:45 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/16 03:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:45:45 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:45:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/16 03:45:51 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:45:56 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:46:01 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:46:06 Waiting for app: TestEnoentGet to get deployed
2021/12/16 03:46:06 App: TestEnoentGet got deployed
2021/12/16 03:46:06 Starting bucket ops to source bucket
2021/12/16 03:46:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:46:11 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:46:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:46:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:46:16 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:46:21 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:46:26 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:46:32 Waiting for app: TestEnoentGet to get un-deployed
2021/12/16 03:46:32 App: TestEnoentGet got un-deployed
2021/12/16 03:46:32 No eventing-con process running
2021/12/16 03:46:32 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/16 03:46:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:46:43 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:46:48 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:46:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:46:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:47:00 Waiting for dst bucket item count to get to: 0 curr count: 65
2021/12/16 03:47:05 Waiting for dst bucket item count to get to: 0 curr count: 65
2021/12/16 03:47:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.39s)
=== RUN   TestEnoentDelete
2021/12/16 03:47:05 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:47:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:47:11 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:47:16 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:47:21 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:47:26 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:47:26 App: TestEnoentDelete got deployed
2021/12/16 03:47:26 Starting bucket ops to source bucket
2021/12/16 03:47:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:47:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:47:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:47:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:47:37 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:47:42 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:47:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:47:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:47:52 App: TestEnoentDelete got un-deployed
2021/12/16 03:47:52 No eventing-con process running
2021/12/16 03:47:52 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 03:47:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:48:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:48:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:48:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:48:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:48:20 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/16 03:48:25 Waiting for dst bucket item count to get to: 0 curr count: 72
2021/12/16 03:48:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:48:25 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:48:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 03:48:31 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:48:36 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:48:41 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:48:46 Waiting for app: TestEnoentDelete to get deployed
2021/12/16 03:48:46 App: TestEnoentDelete got deployed
2021/12/16 03:48:46 Starting bucket ops to source bucket
2021/12/16 03:48:52 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:48:52 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:48:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:48:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:48:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:49:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:49:07 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:49:12 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/16 03:49:12 App: TestEnoentDelete got un-deployed
2021/12/16 03:49:12 No eventing-con process running
2021/12/16 03:49:12 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/16 03:49:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:49:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:49:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:49:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:49:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:49:40 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 03:49:45 Waiting for dst bucket item count to get to: 0 curr count: 79
2021/12/16 03:49:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.28s)
=== RUN   TestError
2021/12/16 03:49:55 Waiting for app: TestError to get un-deployed
2021/12/16 03:49:55 App: TestError got un-deployed
2021/12/16 03:49:55 No eventing-con process running
2021/12/16 03:49:55 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/16 03:49:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:50:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:50:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:50:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:50:09 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:50:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/16 03:50:15 Waiting for app: TestError to get deployed
2021/12/16 03:50:20 Waiting for app: TestError to get deployed
2021/12/16 03:50:25 Waiting for app: TestError to get deployed
2021/12/16 03:50:30 Waiting for app: TestError to get deployed
2021/12/16 03:50:30 App: TestError got deployed
2021/12/16 03:50:30 Starting bucket ops to source bucket
2021/12/16 03:50:35 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/16 03:50:40 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/16 03:50:40 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/16 03:50:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:50:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:50:46 Waiting for app: TestError to get un-deployed
2021/12/16 03:50:51 Waiting for app: TestError to get un-deployed
2021/12/16 03:50:56 Waiting for app: TestError to get un-deployed
2021/12/16 03:51:01 Waiting for app: TestError to get un-deployed
2021/12/16 03:51:01 App: TestError got un-deployed
2021/12/16 03:51:01 No eventing-con process running
2021/12/16 03:51:01 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/16 03:51:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:51:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:51:17 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 03:51:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:51:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:51:29 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/16 03:51:34 Waiting for dst bucket item count to get to: 0 curr count: 156
2021/12/16 03:51:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.84s)
=== RUN   TestCRLF
2021/12/16 03:51:44 Waiting for app: TestCRLF to get un-deployed
2021/12/16 03:51:44 App: TestCRLF got un-deployed
2021/12/16 03:51:44 No eventing-con process running
2021/12/16 03:51:44 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/16 03:51:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:51:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:51:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:51:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:51:58 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:51:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/16 03:52:04 Waiting for app: TestCRLF to get deployed
2021/12/16 03:52:09 Waiting for app: TestCRLF to get deployed
2021/12/16 03:52:14 Waiting for app: TestCRLF to get deployed
2021/12/16 03:52:19 Waiting for app: TestCRLF to get deployed
2021/12/16 03:52:19 App: TestCRLF got deployed
2021/12/16 03:52:19 Starting bucket ops to source bucket
2021/12/16 03:52:24 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 03:52:29 Waiting for dst bucket item count to get to: 100 curr count: 55
2021/12/16 03:52:29 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 03:52:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:52:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:52:34 Waiting for app: TestCRLF to get un-deployed
2021/12/16 03:52:39 Waiting for app: TestCRLF to get un-deployed
2021/12/16 03:52:44 Waiting for app: TestCRLF to get un-deployed
2021/12/16 03:52:49 Waiting for app: TestCRLF to get un-deployed
2021/12/16 03:52:49 App: TestCRLF got un-deployed
2021/12/16 03:52:49 No eventing-con process running
2021/12/16 03:52:49 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/16 03:52:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:53:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:53:06 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 03:53:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:53:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:53:18 Waiting for dst bucket item count to get to: 0 curr count: 74
2021/12/16 03:53:23 Waiting for dst bucket item count to get to: 0 curr count: 19
2021/12/16 03:53:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.78s)
=== RUN   TestImportExport
2021/12/16 03:53:33 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:53:33 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 03:53:33 No eventing-con process running
2021/12/16 03:53:33 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/16 03:53:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:53:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:53:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:53:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:53:52 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:53:52 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 03:53:52 No eventing-con process running
2021/12/16 03:53:52 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/16 03:53:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:53:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:54:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:54:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:54:05 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:54:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 03:54:07 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:54:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 03:54:13 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 03:54:18 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 03:54:23 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 03:54:28 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/16 03:54:28 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/16 03:54:33 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 03:54:38 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 03:54:43 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/16 03:54:43 App: TestImportExport_bucket_op_on_update got deployed
2021/12/16 03:54:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:54:48 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:54:53 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:54:58 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:55:03 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:55:03 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 03:55:03 No eventing-con process running
2021/12/16 03:55:03 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 03:55:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:55:08 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:55:13 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:55:18 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:55:23 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:55:23 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 03:55:23 No eventing-con process running
2021/12/16 03:55:23 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 03:55:40 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/16 03:55:40 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/16 03:55:40 No eventing-con process running
2021/12/16 03:55:40 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/16 03:55:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:55:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:55:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:55:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:55:59 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/16 03:55:59 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/16 03:55:59 No eventing-con process running
2021/12/16 03:55:59 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/16 03:56:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:56:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:56:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:56:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.63s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/16 03:56:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:56:23 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 03:56:23 No eventing-con process running
2021/12/16 03:56:23 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/16 03:56:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:56:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:56:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:56:36 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:56:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 03:56:37 Starting bucket ops to source bucket
2021/12/16 03:56:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:56:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:56:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:57:00 Waiting for dst bucket item count to get to: 5000 curr count: 1116
2021/12/16 03:57:05 Waiting for dst bucket item count to get to: 5000 curr count: 1116
2021/12/16 03:57:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 03:57:10 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 03:57:10 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 03:57:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:57:11 Undeploying app: bucket_op_on_update
2021/12/16 03:57:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:57:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:57:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:57:26 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:57:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:57:31 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 03:57:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:57:31 No eventing-con process running
2021/12/16 03:57:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:57:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:58:15 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:58:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 03:58:16 Starting bucket ops to source bucket
2021/12/16 03:58:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:58:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:58:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 03:58:39 Waiting for dst bucket item count to get to: 5000 curr count: 1998
2021/12/16 03:58:44 Waiting for dst bucket item count to get to: 5000 curr count: 1998
2021/12/16 03:58:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 03:58:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 03:58:49 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 03:58:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 03:58:49 Undeploying app: bucket_op_on_update
2021/12/16 03:58:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 03:58:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:58:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:59:04 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:59:09 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 03:59:09 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 03:59:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 03:59:09 No eventing-con process running
2021/12/16 03:59:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 03:59:53 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 03:59:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 03:59:54 Starting bucket ops to source bucket
2021/12/16 04:00:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:00:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:00:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:00:17 Waiting for dst bucket item count to get to: 5000 curr count: 1189
2021/12/16 04:00:22 Waiting for dst bucket item count to get to: 5000 curr count: 3689
2021/12/16 04:00:22 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:00:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 04:00:27 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 04:00:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:00:27 Undeploying app: bucket_op_on_update
2021/12/16 04:00:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:00:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:00:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:00:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:00:48 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:00:48 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 04:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:00:48 No eventing-con process running
2021/12/16 04:00:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:00:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:01:32 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:01:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:01:33 Starting bucket ops to source bucket
2021/12/16 04:01:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:01:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:01:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:01:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:01:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:02:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 04:02:01 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 04:02:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:02:01 Undeploying app: bucket_op_on_update
2021/12/16 04:02:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:02:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:02:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:02:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:02:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:02:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 04:02:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:02:21 No eventing-con process running
2021/12/16 04:02:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:02:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:03:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:03:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:03:06 Starting bucket ops to source bucket
2021/12/16 04:03:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:03:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:03:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:03:29 Waiting for dst bucket item count to get to: 5000 curr count: 2025
2021/12/16 04:03:34 Waiting for dst bucket item count to get to: 5000 curr count: 2025
2021/12/16 04:03:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:03:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/16 04:03:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/16 04:03:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:03:39 Undeploying app: bucket_op_on_update
2021/12/16 04:03:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:03:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:03:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:03:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:03:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/16 04:03:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/16 04:03:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:03:59 No eventing-con process running
2021/12/16 04:04:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:04:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:04:44 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.93s)
=== RUN   TestOnUpdateN1QLOp
2021/12/16 04:04:54 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 04:04:54 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 04:04:54 No eventing-con process running
2021/12/16 04:04:54 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/16 04:04:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:05:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:05:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:05:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:05:07 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:05:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/16 04:05:08 Starting bucket ops to source bucket
2021/12/16 04:05:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:05:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:05:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:05:31 Waiting for dst bucket item count to get to: 5000 curr count: 19
2021/12/16 04:05:36 Waiting for dst bucket item count to get to: 5000 curr count: 2481
2021/12/16 04:05:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:05:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:05:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:05:42 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 04:05:47 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 04:05:52 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 04:05:57 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/16 04:05:57 App: TestOnUpdateN1QLOp got un-deployed
2021/12/16 04:05:57 No eventing-con process running
2021/12/16 04:05:57 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/16 04:05:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:06:08 Waiting for dst bucket item count to get to: 0 curr count: 4078
2021/12/16 04:06:13 Waiting for dst bucket item count to get to: 0 curr count: 1578
2021/12/16 04:06:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:06:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:06:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:06:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.64s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/16 04:06:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 04:06:35 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 04:06:35 No eventing-con process running
2021/12/16 04:06:35 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/16 04:06:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:06:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:06:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:06:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:06:49 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:06:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:06:50 Starting bucket ops to source bucket
2021/12/16 04:06:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:07:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:07:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:07:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:07:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 04:07:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:07:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:07:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:07:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 04:07:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 04:07:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 04:07:38 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/16 04:07:38 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/16 04:07:38 No eventing-con process running
2021/12/16 04:07:38 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/16 04:07:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:07:50 Waiting for dst bucket item count to get to: 0 curr count: 3034
2021/12/16 04:07:55 Waiting for dst bucket item count to get to: 0 curr count: 3034
2021/12/16 04:07:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:07:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:08:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:08:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (101.61s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/16 04:08:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 04:08:17 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 04:08:17 No eventing-con process running
2021/12/16 04:08:17 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 04:08:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:08:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:08:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:08:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:08:30 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:08:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:08:32 Starting bucket ops to source bucket
2021/12/16 04:08:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:08:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:08:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:08:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:08:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:08:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:08:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:09:00 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 04:09:05 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 04:09:10 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 04:09:15 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 04:09:15 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 04:09:15 No eventing-con process running
2021/12/16 04:09:15 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 04:09:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:09:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:09:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:09:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:09:38 Waiting for dst bucket item count to get to: 0 curr count: 4059
2021/12/16 04:09:43 Waiting for dst bucket item count to get to: 0 curr count: 1559
2021/12/16 04:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (96.48s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/16 04:09:53 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 04:09:53 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 04:09:53 No eventing-con process running
2021/12/16 04:09:53 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/16 04:09:55 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:10:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:10:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:10:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:10:07 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:10:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/16 04:10:08 Starting bucket ops to source bucket
2021/12/16 04:10:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:10:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:10:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:10:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:10:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:10:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:10:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:10:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:10:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 04:10:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 04:10:51 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 04:10:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/16 04:10:56 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/16 04:10:56 No eventing-con process running
2021/12/16 04:10:56 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/16 04:10:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:11:08 Waiting for dst bucket item count to get to: 0 curr count: 3668
2021/12/16 04:11:13 Waiting for dst bucket item count to get to: 0 curr count: 1168
2021/12/16 04:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:11:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:11:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (101.73s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/16 04:11:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 04:11:35 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 04:11:35 No eventing-con process running
2021/12/16 04:11:35 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/16 04:11:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:11:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:11:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:11:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:11:49 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:11:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/16 04:11:50 Starting bucket ops to source bucket
2021/12/16 04:11:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:12:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:12:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:12:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:12:18 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 04:12:18 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:12:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:12:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:12:24 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 04:12:29 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 04:12:34 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 04:12:39 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/16 04:12:39 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/16 04:12:39 No eventing-con process running
2021/12/16 04:12:39 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/16 04:12:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:12:51 Waiting for dst bucket item count to get to: 0 curr count: 4008
2021/12/16 04:12:56 Waiting for dst bucket item count to get to: 0 curr count: 4008
2021/12/16 04:12:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:12:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:13:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:13:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.26s)
=== RUN   TestOnDeleteBucketOp
2021/12/16 04:13:17 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 04:13:17 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 04:13:17 No eventing-con process running
2021/12/16 04:13:17 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/16 04:13:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:13:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:13:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:13:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:13:31 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:13:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 04:13:32 Starting bucket ops to source bucket
2021/12/16 04:13:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:13:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:13:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:13:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:13:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:13:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:13:58 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 04:14:03 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 04:14:08 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 04:14:13 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/16 04:14:13 App: TestOnDeleteBucketOp got un-deployed
2021/12/16 04:14:13 No eventing-con process running
2021/12/16 04:14:13 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/16 04:14:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:14:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:14:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:14:32 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:14:37 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/16 04:14:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.02s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/16 04:14:47 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 04:14:47 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 04:14:47 No eventing-con process running
2021/12/16 04:14:47 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/16 04:14:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:14:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:14:56 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:15:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:15:01 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:15:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/16 04:15:02 Starting bucket ops to source bucket
2021/12/16 04:15:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:15:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:15:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:15:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:15:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:15:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:15:28 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 04:15:33 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 04:15:38 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 04:15:43 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/16 04:15:43 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/16 04:15:43 No eventing-con process running
2021/12/16 04:15:43 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/16 04:15:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:15:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:15:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:16:02 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:16:07 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/16 04:16:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (89.95s)
=== RUN   TestTimerBucketOp
2021/12/16 04:16:17 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 04:16:17 App: TestTimerBucketOp got un-deployed
2021/12/16 04:16:17 No eventing-con process running
2021/12/16 04:16:17 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/16 04:16:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:16:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:16:31 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:16:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 04:16:32 Starting bucket ops to source bucket
2021/12/16 04:16:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:16:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:16:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:16:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:17:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:17:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:17:10 Waiting for dst bucket item count to get to: 5000 curr count: 2202
2021/12/16 04:17:15 Waiting for dst bucket item count to get to: 5000 curr count: 2202
2021/12/16 04:17:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:17:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:17:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:17:20 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 04:17:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 04:17:30 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 04:17:35 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/16 04:17:35 App: TestTimerBucketOp got un-deployed
2021/12/16 04:17:35 No eventing-con process running
2021/12/16 04:17:35 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/16 04:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:17:47 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:17:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:17:59 Waiting for dst bucket item count to get to: 0 curr count: 4761
2021/12/16 04:18:04 Waiting for dst bucket item count to get to: 0 curr count: 2261
2021/12/16 04:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (116.55s)
=== RUN   TestTimerInPastBucketOp
2021/12/16 04:18:14 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 04:18:14 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 04:18:14 No eventing-con process running
2021/12/16 04:18:14 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/16 04:18:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:18:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:18:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:18:27 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:18:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 04:18:29 Starting bucket ops to source bucket
2021/12/16 04:18:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:18:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:18:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:18:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:18:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:19:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:19:07 Waiting for dst bucket item count to get to: 5000 curr count: 2291
2021/12/16 04:19:12 Waiting for dst bucket item count to get to: 5000 curr count: 4791
2021/12/16 04:19:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:19:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:19:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:19:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 04:19:22 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 04:19:27 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 04:19:32 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/16 04:19:32 App: TestTimerInPastBucketOp got un-deployed
2021/12/16 04:19:32 No eventing-con process running
2021/12/16 04:19:32 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/16 04:19:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:19:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:19:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 04:19:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:19:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:20:01 Waiting for dst bucket item count to get to: 0 curr count: 4166
2021/12/16 04:20:06 Waiting for dst bucket item count to get to: 0 curr count: 4166
2021/12/16 04:20:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (121.72s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/16 04:20:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:20:16 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:20:16 No eventing-con process running
2021/12/16 04:20:16 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/16 04:20:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:20:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:20:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:20:29 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:20:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 04:20:30 Starting bucket ops to source bucket
2021/12/16 04:20:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:20:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:20:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:20:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:20:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:21:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:21:08 Waiting for dst bucket item count to get to: 5000 curr count: 3702
2021/12/16 04:21:13 Waiting for dst bucket item count to get to: 5000 curr count: 3702
2021/12/16 04:21:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:21:18 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 04:21:18 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 04:21:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:21:18 Undeploying app: bucket_op_with_timer
2021/12/16 04:21:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:21:24 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:21:29 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:21:34 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:21:39 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:21:39 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:21:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:21:39 No eventing-con process running
2021/12/16 04:21:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:21:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:22:23 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:22:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 04:22:24 Starting bucket ops to source bucket
2021/12/16 04:22:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:22:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:22:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:22:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:22:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:22:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:23:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:23:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 04:23:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:23:12 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 04:23:12 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 04:23:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:23:12 Undeploying app: bucket_op_with_timer
2021/12/16 04:23:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:23:17 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:23:22 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:23:27 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:23:32 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:23:32 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:23:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:23:32 No eventing-con process running
2021/12/16 04:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:23:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:24:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:24:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 04:24:17 Starting bucket ops to source bucket
2021/12/16 04:24:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:24:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:25:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 04:25:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 04:25:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:25:00 Undeploying app: bucket_op_with_timer
2021/12/16 04:25:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:25:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:25:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:25:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:25:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:25:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:25:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:25:20 No eventing-con process running
2021/12/16 04:25:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:25:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:26:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:26:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 04:26:05 Starting bucket ops to source bucket
2021/12/16 04:26:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:26:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:26:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:26:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:26:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:26:38 Waiting for dst bucket item count to get to: 5000 curr count: 1644
2021/12/16 04:26:43 Waiting for dst bucket item count to get to: 5000 curr count: 1644
2021/12/16 04:26:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:26:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 04:26:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 04:26:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:26:49 Undeploying app: bucket_op_with_timer
2021/12/16 04:26:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:26:54 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:26:59 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:27:04 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:27:09 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:27:09 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:27:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:27:09 No eventing-con process running
2021/12/16 04:27:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:27:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:27:53 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:27:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 04:27:54 Starting bucket ops to source bucket
2021/12/16 04:28:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:28:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 04:28:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:28:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/16 04:28:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/16 04:28:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:28:42 Undeploying app: bucket_op_with_timer
2021/12/16 04:28:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:28:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:28:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:28:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:29:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/16 04:29:02 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/16 04:29:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:29:02 No eventing-con process running
2021/12/16 04:29:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:29:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:29:46 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (580.62s)
=== RUN   TestMultipleHandlers
2021/12/16 04:29:56 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 04:29:56 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 04:29:56 No eventing-con process running
2021/12/16 04:29:56 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/16 04:29:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:30:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:30:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:30:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:30:15 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 04:30:15 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 04:30:15 No eventing-con process running
2021/12/16 04:30:15 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/16 04:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:30:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:30:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:30:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:30:28 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:30:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 04:30:30 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:30:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 04:30:36 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 04:30:41 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 04:30:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 04:30:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/16 04:30:51 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/16 04:30:56 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 04:31:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 04:31:06 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/16 04:31:06 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/16 04:31:06 Starting bucket ops to source bucket
2021/12/16 04:31:16 Waiting for dst bucket item count to get to: 5000 curr count: 709
2021/12/16 04:31:21 Waiting for dst bucket item count to get to: 5000 curr count: 2510
2021/12/16 04:31:26 Waiting for dst bucket item count to get to: 5000 curr count: 4301
2021/12/16 04:31:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:31:26 Starting bucket ops to source bucket
2021/12/16 04:31:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:31:41 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 04:31:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:31:41 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:31:41 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:31:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:31:46 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 04:31:51 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 04:31:56 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 04:32:01 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/16 04:32:01 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/16 04:32:01 No eventing-con process running
2021/12/16 04:32:01 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/16 04:32:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:32:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:32:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:32:20 Waiting for dst bucket item count to get to: 0 curr count: 5566
2021/12/16 04:32:25 Waiting for dst bucket item count to get to: 0 curr count: 5566
2021/12/16 04:32:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:32:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:32:30 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 04:32:35 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 04:32:40 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 04:32:45 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/16 04:32:45 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/16 04:32:45 No eventing-con process running
2021/12/16 04:32:45 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/16 04:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:32:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (192.66s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/16 04:33:09 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 04:33:09 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 04:33:09 No eventing-con process running
2021/12/16 04:33:09 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/16 04:33:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:33:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:33:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:33:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:33:22 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:33:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:33:24 Starting bucket ops to source bucket
2021/12/16 04:33:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:33:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:33:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:33:47 Waiting for dst bucket item count to get to: 5000 curr count: 2099
2021/12/16 04:33:52 Waiting for dst bucket item count to get to: 5000 curr count: 4599
2021/12/16 04:33:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:33:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:33:57 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 04:33:57 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 04:33:57 Pausing app: bucket_op_on_update
2021/12/16 04:33:57 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:34:02 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 04:34:02 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 04:34:02 Resuming app: bucket_op_on_update
2021/12/16 04:34:03 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:34:03 Starting bucket ops to source bucket
2021/12/16 04:34:11 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:34:16 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:34:21 Waiting for dst bucket item count to get to: 10000 curr count: 5295
2021/12/16 04:34:26 Waiting for dst bucket item count to get to: 10000 curr count: 7795
2021/12/16 04:34:26 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 04:34:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:34:31 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 04:34:31 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 04:34:31 Pausing app: bucket_op_on_update
2021/12/16 04:34:31 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:34:36 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 04:34:36 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 04:34:36 Resuming app: bucket_op_on_update
2021/12/16 04:34:37 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:34:37 Starting bucket ops to source bucket
2021/12/16 04:34:45 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:34:50 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:34:55 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/16 04:34:55 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 04:34:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:35:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 04:35:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 04:35:00 Pausing app: bucket_op_on_update
2021/12/16 04:35:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:35:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 04:35:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 04:35:06 Resuming app: bucket_op_on_update
2021/12/16 04:35:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:35:07 Starting bucket ops to source bucket
2021/12/16 04:35:15 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:35:20 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:35:25 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/16 04:35:25 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 04:35:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:35:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 04:35:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 04:35:30 Pausing app: bucket_op_on_update
2021/12/16 04:35:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:35:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 04:35:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 04:35:35 Resuming app: bucket_op_on_update
2021/12/16 04:35:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:35:36 Starting bucket ops to source bucket
2021/12/16 04:35:44 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:35:49 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:35:54 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:35:54 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 04:35:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:35:59 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/16 04:35:59 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/16 04:35:59 Pausing app: bucket_op_on_update
2021/12/16 04:35:59 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:36:04 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/16 04:36:04 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/16 04:36:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:36:09 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 04:36:14 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 04:36:19 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 04:36:24 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/16 04:36:24 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/16 04:36:24 No eventing-con process running
2021/12/16 04:36:24 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/16 04:36:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:36:36 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 04:36:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:36:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:36:48 Waiting for dst bucket item count to get to: 0 curr count: 16906
2021/12/16 04:36:53 Waiting for dst bucket item count to get to: 0 curr count: 4406
2021/12/16 04:36:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.14s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/16 04:37:03 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 04:37:03 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 04:37:03 No eventing-con process running
2021/12/16 04:37:03 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/16 04:37:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:37:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:37:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:37:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:37:17 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:37:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 04:37:18 Starting bucket ops to source bucket
2021/12/16 04:37:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:37:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:37:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:37:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:37:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:37:51 Waiting for dst bucket item count to get to: 5000 curr count: 1120
2021/12/16 04:37:56 Waiting for dst bucket item count to get to: 5000 curr count: 1120
2021/12/16 04:37:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:37:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:38:01 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 04:38:01 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 04:38:01 Pausing app: bucket_op_with_timer
2021/12/16 04:38:01 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:38:06 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 04:38:06 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 04:38:06 Resuming app: bucket_op_with_timer
2021/12/16 04:38:07 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:38:07 Starting bucket ops to source bucket
2021/12/16 04:38:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:38:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:38:25 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:38:30 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:38:35 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:38:40 Waiting for dst bucket item count to get to: 10000 curr count: 6475
2021/12/16 04:38:45 Waiting for dst bucket item count to get to: 10000 curr count: 8975
2021/12/16 04:38:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 04:38:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:38:50 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 04:38:50 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 04:38:50 Pausing app: bucket_op_with_timer
2021/12/16 04:38:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:38:55 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 04:38:55 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 04:38:55 Resuming app: bucket_op_with_timer
2021/12/16 04:38:56 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:38:56 Starting bucket ops to source bucket
2021/12/16 04:39:04 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:39:09 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:39:14 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:39:19 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:39:24 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/16 04:39:29 Waiting for dst bucket item count to get to: 15000 curr count: 12367
2021/12/16 04:39:34 Waiting for dst bucket item count to get to: 15000 curr count: 14867
2021/12/16 04:39:34 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/16 04:39:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:39:39 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 04:39:40 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 04:39:40 Pausing app: bucket_op_with_timer
2021/12/16 04:39:40 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:39:45 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 04:39:45 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 04:39:45 Resuming app: bucket_op_with_timer
2021/12/16 04:39:46 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:39:46 Starting bucket ops to source bucket
2021/12/16 04:39:54 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:39:59 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:40:04 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:40:09 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:40:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/16 04:40:19 Waiting for dst bucket item count to get to: 20000 curr count: 17845
2021/12/16 04:40:24 Waiting for dst bucket item count to get to: 20000 curr count: 17845
2021/12/16 04:40:24 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 04:40:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:40:29 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 04:40:29 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 04:40:29 Pausing app: bucket_op_with_timer
2021/12/16 04:40:29 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:40:34 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 04:40:34 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 04:40:34 Resuming app: bucket_op_with_timer
2021/12/16 04:40:35 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:40:35 Starting bucket ops to source bucket
2021/12/16 04:40:43 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:40:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:40:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:40:58 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:41:03 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/16 04:41:08 Waiting for dst bucket item count to get to: 25000 curr count: 23648
2021/12/16 04:41:13 Waiting for dst bucket item count to get to: 25000 curr count: 23648
2021/12/16 04:41:13 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 04:41:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:41:18 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/16 04:41:18 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/16 04:41:18 Pausing app: bucket_op_with_timer
2021/12/16 04:41:18 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:41:23 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/16 04:41:23 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/16 04:41:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:41:28 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 04:41:33 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 04:41:38 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 04:41:43 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/16 04:41:43 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/16 04:41:43 No eventing-con process running
2021/12/16 04:41:43 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/16 04:41:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:41:55 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 04:41:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:41:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:42:07 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 04:42:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (314.00s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/16 04:42:17 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 04:42:17 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 04:42:17 No eventing-con process running
2021/12/16 04:42:17 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/16 04:42:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:42:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:42:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:42:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:42:31 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:42:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/16 04:42:37 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 04:42:42 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 04:42:47 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 04:42:52 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/16 04:42:52 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/16 04:42:52 Starting bucket ops to source bucket
2021/12/16 04:43:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:43:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:43:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:43:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:43:21 Waiting for dst bucket item count to get to: 5000 curr count: 827
2021/12/16 04:43:26 Waiting for dst bucket item count to get to: 5000 curr count: 827
2021/12/16 04:43:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:43:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:43:26 Pausing app: bucket_op_with_timer
2021/12/16 04:43:26 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:43:31 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/16 04:43:31 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/16 04:43:31 Resuming app: bucket_op_with_timer
2021/12/16 04:43:32 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:43:32 Starting bucket ops to source bucket
2021/12/16 04:43:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:43:49 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:43:54 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:43:59 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:44:04 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:44:09 Waiting for dst bucket item count to get to: 10000 curr count: 6865
2021/12/16 04:44:14 Waiting for dst bucket item count to get to: 10000 curr count: 6865
2021/12/16 04:44:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 04:44:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:44:19 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 04:44:24 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 04:44:29 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 04:44:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/16 04:44:34 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/16 04:44:34 No eventing-con process running
2021/12/16 04:44:34 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/16 04:44:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:44:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 04:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:44:48 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:44:58 Waiting for dst bucket item count to get to: 0 curr count: 6716
2021/12/16 04:45:03 Waiting for dst bucket item count to get to: 0 curr count: 1716
2021/12/16 04:45:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (175.99s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/16 04:45:13 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 04:45:13 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 04:45:13 No eventing-con process running
2021/12/16 04:45:13 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/16 04:45:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:45:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:45:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:45:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:45:27 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:45:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 04:45:33 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 04:45:38 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 04:45:43 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 04:45:48 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/16 04:45:48 App: TestPauseResumeWithEventingReb got deployed
2021/12/16 04:45:48 Starting bucket ops to source bucket
2021/12/16 04:45:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:46:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:46:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:46:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:46:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:46:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:46:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:46:17 Pausing app: bucket_op_with_timer
2021/12/16 04:46:17 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:46:22 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/16 04:46:22 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/16 04:46:22 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 04:46:23 Removing node(s): [] from the cluster
2021/12/16 04:46:28 Rebalance progress: 29.16666666666666
2021/12/16 04:46:33 Rebalance progress: 29.16666666666666
2021/12/16 04:46:38 Rebalance progress: 100
2021/12/16 04:46:38 Resuming app: bucket_op_with_timer
2021/12/16 04:46:39 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:46:39 Starting bucket ops to source bucket
2021/12/16 04:46:52 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:46:57 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:47:02 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:47:07 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:47:12 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/16 04:47:17 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/16 04:47:17 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 04:47:17 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 04:47:22 Rebalance progress: 29.16666666666666
2021/12/16 04:47:27 Rebalance progress: 29.16666666666666
2021/12/16 04:47:32 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 04:47:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:47:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:48:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:48:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:48:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:48:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:49:07 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:49:22 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:49:37 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 04:49:52 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:70} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: e562dcbdef9639cbfd3c3c8e77ac6d3e
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: b4311bdbefca268e59ce409a7b4c096a
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
Producer node: 127.0.0.1:9302	Node UUID: 6623cdd7a41158e9dbf60c8ca2c46105
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]


2021/12/16 04:50:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:50:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 04:50:12 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 04:50:17 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 04:50:22 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 04:50:27 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/16 04:50:27 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/16 04:50:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:50:27 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/16 04:50:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:50:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9855
2021/12/16 04:50:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4855
2021/12/16 04:50:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:50:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:50:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 04:50:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (352.67s)
=== RUN   TestBucketDeleteAfterPause
2021/12/16 04:51:06 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 04:51:06 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 04:51:06 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:51:06 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/16 04:51:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:51:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:51:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:51:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:51:19 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:51:20 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/16 04:51:25 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 04:51:30 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 04:51:35 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 04:51:40 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/16 04:51:40 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/16 04:51:40 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 04:51:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:51:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:51:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:51:50 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/16 04:51:50 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:51:55 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/16 04:51:55 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/16 04:51:55 handler_test.go:716: Deleting source bucket: default
2021/12/16 04:52:02 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/16 04:52:02 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/16 04:52:07 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/16 04:52:07 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/16 04:52:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:52:07 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/16 04:52:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:52:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 04:52:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:52:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:52:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4098
2021/12/16 04:52:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4098
2021/12/16 04:52:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (99.90s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/16 04:52:46 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 04:52:46 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 04:52:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:52:46 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/16 04:52:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:52:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:52:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:52:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:52:59 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:53:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 04:53:05 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 04:53:10 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 04:53:15 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 04:53:20 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/16 04:53:20 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/16 04:53:20 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 04:53:55 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/16 04:53:55 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:54:00 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/16 04:54:00 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/16 04:57:00 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/16 04:57:00 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/16 04:57:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/16 04:57:06 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 04:57:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:57:06 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/16 04:57:11 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 04:57:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:57:11 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3124 expected 5000
2021/12/16 04:57:16 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 04:57:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:57:16 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3904 expected 5000
2021/12/16 04:57:21 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/16 04:57:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:57:21 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/16 04:57:21 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/16 04:57:21 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:57:31 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 04:57:36 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 04:57:41 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/16 04:57:41 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/16 04:57:41 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:57:41 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/16 04:57:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:57:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 04:57:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 04:57:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:57:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:58:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (328.89s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/16 04:58:14 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 04:58:14 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 04:58:14 eventingSetup.go:46: No eventing-con process running
2021/12/16 04:58:14 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/16 04:58:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:58:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:58:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 04:58:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 04:58:28 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 04:58:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 04:58:34 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:58:39 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:58:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:58:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:58:49 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 04:58:49 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 04:58:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 04:59:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 320
2021/12/16 04:59:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2820
2021/12/16 04:59:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 04:59:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 04:59:08 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 04:59:08 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 04:59:38 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/16 04:59:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 04:59:44 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:59:49 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:59:54 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:59:59 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/16 04:59:59 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/16 04:59:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:00:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:00:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 05:00:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:00:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:00:17 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/16 05:00:17 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 05:00:27 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 05:00:32 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 05:00:37 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/16 05:00:37 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/16 05:00:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:00:37 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/16 05:00:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:00:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:00:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:00:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.68s)
=== RUN   TestCPPWorkerCleanup
2021/12/16 05:01:00 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 05:01:00 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 05:01:00 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:01:00 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/16 05:01:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:01:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:01:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:01:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:01:13 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:01:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/16 05:01:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:01:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:01:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:01:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:01:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 10
2021/12/16 05:01:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1340
2021/12/16 05:01:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3830
2021/12/16 05:01:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:01:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:01:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:01:52 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 05:01:57 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 05:02:02 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 05:02:07 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/16 05:02:07 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/16 05:02:07 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:02:07 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/16 05:02:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:02:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3575
2021/12/16 05:02:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1075
2021/12/16 05:02:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:02:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:02:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:02:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.68s)
=== RUN   TestWithUserXattrs
2021/12/16 05:03:15 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 05:03:15 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 05:03:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:03:15 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/16 05:03:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:03:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:03:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:03:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:03:28 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:03:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:03:32 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:03:33 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/16 05:03:38 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 05:03:43 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 05:03:48 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 05:03:53 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/16 05:03:53 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/16 05:03:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:04:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:04:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2399
2021/12/16 05:04:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4899
2021/12/16 05:04:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:04:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:04:13 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:04:18 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 05:04:23 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 05:04:28 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 05:04:33 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/16 05:04:33 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/16 05:04:33 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:04:33 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/16 05:04:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:04:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:04:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 05:04:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:04:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:05:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4330
2021/12/16 05:05:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4330
2021/12/16 05:05:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (120.86s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/16 05:05:16 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:05:16 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 05:05:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:05:16 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/16 05:05:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:05:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:05:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:05:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:05:29 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:05:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/16 05:05:35 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 05:05:40 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 05:05:45 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 05:05:50 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/16 05:05:50 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/16 05:05:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:05:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:06:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:06:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/16 05:06:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:06:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:06:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:06:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:06:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 05:06:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:06:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:06:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2814
2021/12/16 05:06:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2814
2021/12/16 05:06:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:06:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:07:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16577
2021/12/16 05:07:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 16577
2021/12/16 05:07:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/16 05:07:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:07:15 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 05:07:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:07:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:07:30 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:07:35 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:07:35 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 05:07:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:07:40 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/16 05:07:40 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/16 05:07:40 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:07:40 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/16 05:07:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:07:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 05:07:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/16 05:07:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:07:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:08:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 20224
2021/12/16 05:08:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7724
2021/12/16 05:08:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (187.67s)
=== RUN   TestMetaBucketDelete
2021/12/16 05:08:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 05:08:23 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 05:08:23 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:08:23 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/16 05:08:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:08:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:08:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:08:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:08:36 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:08:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 05:08:43 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 05:08:48 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 05:08:53 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 05:08:58 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/16 05:08:58 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/16 05:08:58 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/16 05:08:59 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/16 05:09:04 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 05:09:04 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 05:09:19 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/16 05:09:19 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/16 05:09:19 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:09:19 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/16 05:09:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:09:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:09:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:09:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.68s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/16 05:09:42 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:09:42 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:09:42 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:09:42 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 05:09:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:09:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:09:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:09:55 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:09:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 05:10:15 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/16 05:10:16 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/16 05:10:31 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:10:31 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:10:46 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:10:46 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:10:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:10:46 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 05:10:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:10:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:10:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:10:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.88s)
=== RUN   TestSourceBucketDelete
2021/12/16 05:11:09 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 05:11:09 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 05:11:09 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:11:09 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/16 05:11:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:11:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:11:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:11:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:11:22 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:11:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/16 05:11:28 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 05:11:33 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 05:11:38 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 05:11:43 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/16 05:11:43 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/16 05:11:43 handler_test.go:945: Deleting source bucket: default
2021/12/16 05:11:45 handler_test.go:947: Deleted source bucket: default
2021/12/16 05:11:50 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 05:11:50 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 05:12:05 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/16 05:12:05 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/16 05:12:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:12:05 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/16 05:12:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:12:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:12:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:12:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.85s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/16 05:12:28 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:12:28 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:12:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:12:28 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 05:12:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:12:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:12:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:12:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:12:41 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:12:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 05:13:01 handler_test.go:966: Deleting source bucket: default
2021/12/16 05:13:02 handler_test.go:968: Deleted source bucket: default
2021/12/16 05:13:17 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:13:17 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:13:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:13:32 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:13:32 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:13:32 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 05:13:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:13:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:13:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:13:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.68s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/16 05:13:55 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:13:55 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:13:55 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:13:55 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/16 05:13:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:14:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:14:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:14:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:14:09 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:14:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 05:14:28 handler_test.go:988: Deleting source bucket: default
2021/12/16 05:14:30 handler_test.go:990: Deleted source bucket: default
2021/12/16 05:14:30 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/16 05:14:30 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/16 05:14:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:14:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:15:00 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/16 05:15:00 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/16 05:15:00 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:15:01 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/16 05:15:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:15:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:15:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:15:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.12s)
=== RUN   TestUndeployDuringBootstrap
2021/12/16 05:15:24 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 05:15:24 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 05:15:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:15:24 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/16 05:15:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:15:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:15:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:15:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:15:37 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:15:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 05:15:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:15:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:15:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:15:50 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 05:15:50 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/16 05:15:50 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/16 05:15:55 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/16 05:16:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 05:16:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 05:16:10 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 05:16:10 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 05:16:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:16:35 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/16 05:16:35 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/16 05:16:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:16:35 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/16 05:16:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:16:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:16:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:16:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:16:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3037
2021/12/16 05:17:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 537
2021/12/16 05:17:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.59s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/16 05:17:08 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:17:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/16 05:17:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 05:17:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 05:17:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 05:17:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/16 05:17:29 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/16 05:17:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/16 05:17:29 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 05:17:30 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/16 05:17:35 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/16 05:17:35 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/16 05:17:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:17:35 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/16 05:17:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:17:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:17:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:17:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.59s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/16 05:17:58 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:17:58 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 05:17:58 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:17:58 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/16 05:17:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:18:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:18:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:18:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:18:11 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:18:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/16 05:18:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 05:18:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 05:18:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 05:18:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/16 05:18:32 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/16 05:18:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:19:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:19:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:19:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:19:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:19:22 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:19:22 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 05:19:23 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:21:08 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/16 05:21:08 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/16 05:21:08 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:21:08 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/16 05:21:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:21:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:21:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 05:21:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:21:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:21:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:21:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.05s)
=== RUN   TestUndeployWithKVFailover
2021/12/16 05:21:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 05:21:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/16 05:21:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:21:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/16 05:21:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:21:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:21:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:21:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:21:59 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:22:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/16 05:22:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 05:22:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:22:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:22:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:22:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:22:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:22:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 5
2021/12/16 05:22:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3463
2021/12/16 05:22:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3463
2021/12/16 05:22:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:22:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 05:22:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 05:22:48 Rebalance progress: 29.16666666666666
2021/12/16 05:22:53 Rebalance progress: 29.16666666666666
2021/12/16 05:22:58 Rebalance progress: 100
2021/12/16 05:22:58 Failing over: [127.0.0.1:19003]
2021/12/16 05:23:08 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/16 05:24:08 Removing node(s): [] from the cluster
2021/12/16 05:24:13 Rebalance progress: 29.16666666666666
2021/12/16 05:24:18 Rebalance progress: 29.16666666666666
2021/12/16 05:24:23 Rebalance progress: 100
2021/12/16 05:24:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 05:24:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 05:24:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:24:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/16 05:24:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/16 05:24:33 No eventing-con process running
2021/12/16 05:24:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/16 05:24:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:24:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:24:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 05:24:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:24:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:25:01 Waiting for dst bucket item count to get to: 0 curr count: 4772
2021/12/16 05:25:06 Waiting for dst bucket item count to get to: 0 curr count: 4772
2021/12/16 05:25:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.23s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/16 05:25:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 05:25:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 05:25:16 No eventing-con process running
2021/12/16 05:25:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/16 05:25:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:25:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:25:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:25:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:25:29 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:25:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/16 05:25:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 05:25:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 05:25:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 05:25:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/16 05:25:50 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/16 05:25:50 Starting bucket ops to source bucket
2021/12/16 05:26:11 Waiting for dst bucket item count to get to: 20000 curr count: 9986
2021/12/16 05:26:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/16 05:26:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:26:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:26:23 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/16 05:26:28 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 05:26:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:26:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:26:40 Waiting for dst bucket item count to get to: 0 curr count: 14031
2021/12/16 05:26:45 Waiting for dst bucket item count to get to: 0 curr count: 14031
2021/12/16 05:26:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:26:45 Starting bucket ops to source bucket
2021/12/16 05:26:58 Waiting for dst bucket item count to get to: 10000 curr count: 3838
2021/12/16 05:27:03 Waiting for dst bucket item count to get to: 10000 curr count: 3838
2021/12/16 05:27:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 05:27:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:27:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:27:08 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 05:27:13 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 05:27:18 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 05:27:23 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/16 05:27:23 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/16 05:27:23 No eventing-con process running
2021/12/16 05:27:23 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/16 05:27:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:27:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 05:27:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:27:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:27:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 05:27:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.31s)
=== RUN   TestTimerOverwrite
2021/12/16 05:27:56 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 05:27:56 App: TestTimerOverwrite got un-deployed
2021/12/16 05:27:56 No eventing-con process running
2021/12/16 05:27:56 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/16 05:27:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:28:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:28:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:28:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:28:09 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:28:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/16 05:28:16 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 05:28:21 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 05:28:26 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 05:28:31 Waiting for app: TestTimerOverwrite to get deployed
2021/12/16 05:28:31 App: TestTimerOverwrite got deployed
2021/12/16 05:28:41 Starting bucket ops to source bucket
2021/12/16 05:28:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:28:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 05:29:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:29:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:29:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:30:00 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 05:30:05 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 05:30:10 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 05:30:15 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/16 05:30:15 App: TestTimerOverwrite got un-deployed
2021/12/16 05:30:15 No eventing-con process running
2021/12/16 05:30:15 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/16 05:30:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:30:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:30:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:30:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:30:38 Waiting for dst bucket item count to get to: 0 curr count: 4288
2021/12/16 05:30:43 Waiting for dst bucket item count to get to: 0 curr count: 1827
2021/12/16 05:30:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (177.17s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/16 05:30:48 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 05:30:49 Removing node(s): [] from the cluster
2021/12/16 05:30:54 Rebalance progress: 100
2021/12/16 05:30:54 Failing over: [127.0.0.1:19003]
2021/12/16 05:30:59 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 05:30:59 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 05:30:59 No eventing-con process running
2021/12/16 05:30:59 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 05:31:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:31:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:31:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:31:12 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:31:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/16 05:31:13 Starting bucket ops to source bucket
2021/12/16 05:31:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:31:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:31:23 Context: Post to set retry counter, response code: 200 dump: 
2021/12/16 05:31:23 set retry counter, response &{[] }
2021/12/16 05:32:28 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 05:32:28 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 05:32:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:32:28 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/16 05:32:33 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/16 05:32:33 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/16 05:32:33 No eventing-con process running
2021/12/16 05:32:33 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/16 05:32:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:32:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:32:56 Waiting for dst bucket item count to get to: 0 curr count: 2960
2021/12/16 05:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:32:56 Removing node(s): [] from the cluster
2021/12/16 05:33:01 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (137.89s)
=== RUN   TestOnUpdateSrcMutation
2021/12/16 05:33:11 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 05:33:11 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 05:33:11 No eventing-con process running
2021/12/16 05:33:11 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/16 05:33:13 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:33:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:33:24 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:33:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 05:33:26 Starting bucket ops to source bucket
2021/12/16 05:33:53 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 05:33:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:33:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:33:58 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 05:34:03 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 05:34:08 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 05:34:13 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/16 05:34:13 App: TestOnUpdateSrcMutation got un-deployed
2021/12/16 05:34:13 No eventing-con process running
2021/12/16 05:34:13 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/16 05:34:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:34:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 05:34:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:34:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:34:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:34:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (94.51s)
=== RUN   TestOnDeleteSrcMutation
2021/12/16 05:34:46 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 05:34:46 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 05:34:46 No eventing-con process running
2021/12/16 05:34:46 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/16 05:34:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:34:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:34:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:34:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:34:59 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:35:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/16 05:35:00 Starting bucket ops to source bucket
2021/12/16 05:35:29 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:35:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:35:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:35:34 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 05:35:39 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 05:35:44 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 05:35:49 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/16 05:35:49 App: TestOnDeleteSrcMutation got un-deployed
2021/12/16 05:35:49 No eventing-con process running
2021/12/16 05:35:49 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/16 05:35:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:36:00 Waiting for dst bucket item count to get to: 0 curr count: 4342
2021/12/16 05:36:05 Waiting for dst bucket item count to get to: 0 curr count: 4342
2021/12/16 05:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:36:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:36:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (96.25s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/16 05:36:22 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 05:36:22 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 05:36:22 No eventing-con process running
2021/12/16 05:36:22 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/16 05:36:23 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:36:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:36:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:36:35 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:36:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 05:36:36 Starting bucket ops to source bucket
2021/12/16 05:37:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 05:37:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:37:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:37:18 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 05:37:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 05:37:28 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 05:37:34 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/16 05:37:34 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/16 05:37:34 No eventing-con process running
2021/12/16 05:37:34 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/16 05:37:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:37:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 05:37:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:37:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:37:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.44s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/16 05:38:02 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 05:38:02 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 05:38:02 No eventing-con process running
2021/12/16 05:38:02 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/16 05:38:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:38:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:38:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:38:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:38:15 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:38:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/16 05:38:16 Starting bucket ops to source bucket
2021/12/16 05:38:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 05:38:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:38:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:39:00 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 05:39:05 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 05:39:10 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 05:39:15 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/16 05:39:15 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/16 05:39:15 No eventing-con process running
2021/12/16 05:39:15 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/16 05:39:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:39:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 05:39:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:39:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:39:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.34s)
=== RUN   TestInterHandlerRecursion
2021/12/16 05:39:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:39:43 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 05:39:43 No eventing-con process running
2021/12/16 05:39:43 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 05:39:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:39:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:39:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:39:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:40:01 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:40:01 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 05:40:01 No eventing-con process running
2021/12/16 05:40:01 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 05:40:02 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:40:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:40:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:40:14 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:40:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 05:40:15 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:40:20 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:40:25 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:40:30 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:40:35 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:40:35 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 05:40:35 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:40:35 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/16 05:40:35 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/16 05:40:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:40:41 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:40:46 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:40:51 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:40:56 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:40:56 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 05:40:56 No eventing-con process running
2021/12/16 05:40:56 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 05:40:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:41:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:41:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:41:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:41:14 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:41:14 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 05:41:14 No eventing-con process running
2021/12/16 05:41:14 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 05:41:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:41:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:41:22 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:41:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (113.80s)
=== RUN   TestInterBucketRecursion
2021/12/16 05:41:32 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:41:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 05:41:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:41:33 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:41:33 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 05:41:38 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 05:41:43 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 05:41:48 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 05:41:53 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/16 05:41:53 App: TestInterBucketRecursion_function1 got deployed
2021/12/16 05:41:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:41:58 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:42:03 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:42:08 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:42:13 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:42:13 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/16 05:42:13 No eventing-con process running
2021/12/16 05:42:13 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 05:42:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:42:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:42:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:42:31 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/16 05:42:31 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/16 05:42:31 No eventing-con process running
2021/12/16 05:42:31 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/16 05:42:32 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:42:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:42:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:42:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.53s)
=== RUN   TestLargeHandler
2021/12/16 05:42:44 Changed force_compress value to false
2021/12/16 05:42:44 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:42:44 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/16 05:42:44 Changed force_compress value to true
2021/12/16 05:42:44 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:42:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/16 05:42:51 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 05:42:56 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 05:43:01 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 05:43:06 Waiting for app: TestLargeHandler_function to get deployed
2021/12/16 05:43:06 App: TestLargeHandler_function got deployed
2021/12/16 05:43:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:43:11 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 05:43:17 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 05:43:22 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 05:43:27 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/16 05:43:27 App: TestLargeHandler_function got un-deployed
2021/12/16 05:43:27 No eventing-con process running
2021/12/16 05:43:27 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/16 05:43:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:43:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:43:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:43:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.34s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/16 05:43:50 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:43:50 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 05:43:50 No eventing-con process running
2021/12/16 05:43:50 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/16 05:43:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:43:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:44:08 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:44:08 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 05:44:08 No eventing-con process running
2021/12/16 05:44:08 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/16 05:44:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:44:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:44:16 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:44:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:44:21 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:44:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/16 05:44:22 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:44:27 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:44:32 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:44:37 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:44:42 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/16 05:44:42 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/16 05:44:42 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:44:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/16 05:44:43 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:44:48 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 05:44:53 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 05:44:58 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 05:45:03 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/16 05:45:03 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/16 05:45:03 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/16 05:45:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:45:08 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:45:13 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:45:18 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:45:23 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/16 05:45:23 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/16 05:45:23 No eventing-con process running
2021/12/16 05:45:24 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/16 05:45:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:45:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:45:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:45:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:45:43 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:45:48 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:45:53 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:45:58 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/16 05:45:58 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/16 05:45:58 No eventing-con process running
2021/12/16 05:45:58 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/16 05:45:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:46:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:46:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:46:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.23s)
=== RUN   TestAllowInterBucketRecursion
2021/12/16 05:46:16 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:46:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 05:46:17 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:46:22 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 05:46:27 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 05:46:32 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 05:46:37 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/16 05:46:37 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/16 05:46:37 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:46:38 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 05:46:38 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:46:43 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 05:46:48 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 05:46:53 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 05:46:58 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/16 05:46:58 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/16 05:46:58 TestAllowInterBucketRecursion_function2 is deployed
2021/12/16 05:46:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:47:03 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:47:08 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:47:13 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:47:18 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/16 05:47:18 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/16 05:47:18 No eventing-con process running
2021/12/16 05:47:18 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/16 05:47:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:47:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:47:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:47:38 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 05:47:43 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 05:47:48 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 05:47:53 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/16 05:47:53 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/16 05:47:53 No eventing-con process running
2021/12/16 05:47:53 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/16 05:47:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:47:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:48:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:48:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.25s)
=== RUN   TestInterBucketRecursion2
2021/12/16 05:48:12 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:48:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:48:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:48:36 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:48:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 05:48:42 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 05:48:47 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 05:48:52 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 05:48:57 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/16 05:48:57 App: TestInterBucketRecursion2_function1 got deployed
2021/12/16 05:48:57 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:48:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 05:49:03 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 05:49:08 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 05:49:13 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 05:49:18 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/16 05:49:18 App: TestInterBucketRecursion2_function2 got deployed
2021/12/16 05:49:18 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:49:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 05:49:24 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 05:49:29 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 05:49:34 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 05:49:39 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/16 05:49:39 App: TestInterBucketRecursion2_function3 got deployed
2021/12/16 05:49:39 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:49:39 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/16 05:49:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:49:45 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 05:49:50 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 05:49:55 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 05:50:00 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/16 05:50:00 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/16 05:50:00 No eventing-con process running
2021/12/16 05:50:00 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/16 05:50:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:50:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:50:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:50:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:50:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:50:18 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 05:50:23 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 05:50:28 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 05:50:33 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/16 05:50:33 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/16 05:50:33 No eventing-con process running
2021/12/16 05:50:33 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/16 05:50:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:50:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:50:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:50:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:50:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:50:51 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 05:50:56 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 05:51:01 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 05:51:06 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/16 05:51:06 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/16 05:51:07 No eventing-con process running
2021/12/16 05:51:07 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/16 05:51:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:51:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:51:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:51:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:51:25 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/16 05:51:25 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/16 05:51:25 No eventing-con process running
2021/12/16 05:51:25 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/16 05:51:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:51:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:51:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:51:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.83s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/16 05:51:45 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 05:51:50 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/16 05:51:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:52:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:52:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:52:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:52:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:53:08 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:53:23 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:53:38 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/16 05:53:53 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:69} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/16 05:54:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:54:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 05:54:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 05:54:14 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 05:54:14 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:54:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 05:54:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 05:54:19 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/16 05:54:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:54:24 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 05:54:24 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 05:54:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:54:24 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/16 05:54:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:54:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:54:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:54:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:54:42 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:54:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 05:54:48 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 05:54:53 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 05:54:58 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 05:55:03 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/16 05:55:03 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: e562dcbdef9639cbfd3c3c8e77ac6d3e
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
Producer node: 127.0.0.1:9303	Node UUID: b4311bdbefca268e59ce409a7b4c096a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 6623cdd7a41158e9dbf60c8ca2c46105
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]


2021/12/16 05:55:04 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:55:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 05:55:09 Rebalance progress: 29.16666666666666
2021/12/16 05:55:14 Rebalance progress: 29.16666666666666
2021/12/16 05:55:19 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6623cdd7a41158e9dbf60c8ca2c46105
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
Producer node: 172.31.2.134:9300	Node UUID: e562dcbdef9639cbfd3c3c8e77ac6d3e
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: b4311bdbefca268e59ce409a7b4c096a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/16 05:55:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:55:19 handler_test.go:1674: Deleting source bucket: default
2021/12/16 05:55:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 05:55:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 05:55:26 Rebalance progress: 29.16666666666666
2021/12/16 05:55:31 Rebalance progress: 29.16666666666666
2021/12/16 05:55:36 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 903
	vb list dump: [0-1, 3-16, 18-20, 22-26, 28-31, 33-35, 37-39, 42, 44-51, 53-62, 64-67, 69-76, 78-114, 116-146, 148-149, 151, 154-157, 159-164, 166-172, 174-175, 178-198, 200-206, 208, 210-228, 230-233, 235, 237-241, 243-245, 247-251, 254-269, 271-275, 277, 279-292, 294-311, 313-324, 326-343, 346-349, 351-354, 356-357, 359-380, 382-386, 388-393, 396-410, 412-426, 428-435, 437-442, 444-449, 451-457, 460-461, 463-464, 466, 468, 470-475, 477-499, 502, 504-505, 511-577, 579-586, 588-590, 593-602, 604-616, 618-621, 623-625, 627-637, 639-647, 649-654, 656-662, 664-670, 672-696, 698-702, 704-715, 717-727, 729-731, 733-748, 750-767, 769-776, 778-781, 783-784, 786, 788-791, 793, 795-796, 798-801, 803-808, 810-826, 828-830, 832, 834-847, 849-852, 854, 856-857, 859-876, 878-883, 885-909, 911-917, 919-933, 935-936, 938-939, 941-944, 946, 948-951, 953-961, 965-981, 983-992, 994-997, 999-1004, 1006-1023]
	stream status: stopped
	len: 121
	vb list dump: [2, 17, 21, 27, 32, 36, 40-41, 43, 52, 63, 68, 77, 115, 147, 150, 152-153, 158, 165, 173, 176-177, 199, 207, 209, 229, 234, 236, 242, 246, 252-253, 270, 276, 278, 293, 312, 325, 344-345, 350, 355, 358, 381, 387, 394-395, 411, 427, 436, 443, 450, 458-459, 462, 465, 467, 469, 476, 500-501, 503, 506-510, 578, 587, 591-592, 603, 617, 622, 626, 638, 648, 655, 663, 671, 697, 703, 716, 728, 732, 749, 768, 777, 782, 785, 787, 792, 794, 797, 802, 809, 827, 831, 833, 848, 853, 855, 858, 877, 884, 910, 918, 934, 937, 940, 945, 947, 952, 962-964, 982, 993, 998, 1005]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 6623cdd7a41158e9dbf60c8ca2c46105
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0-1, 3-16, 18-20, 22-26, 28-31, 33-35, 37-39, 42, 44-51, 53-62, 64-67, 69-76, 78-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114, 116-146, 148-149, 151, 154-157, 159-164, 166-172, 174-175, 178-198, 200-206, 208, 210-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228, 230-233, 235, 237-241, 243-245, 247-251, 254-269, 271-275, 277, 279-292, 294-311, 313-324, 326-341]
Producer node: 172.31.2.134:9300	Node UUID: e562dcbdef9639cbfd3c3c8e77ac6d3e
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [911-917, 919-933, 935-936, 938-939, 941-944, 946, 948-951, 953-961, 965-981, 983-992, 994-997, 999-1004, 1006-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [683-696, 698-702, 704-715, 717-727, 729-731, 733-748, 750-767, 769-776, 778-781, 783-784, 786, 788-791, 793, 795-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [798-801, 803-808, 810-826, 828-830, 832, 834-847, 849-852, 854, 856-857, 859-876, 878-883, 885-909]
Producer node: 127.0.0.1:9303	Node UUID: b4311bdbefca268e59ce409a7b4c096a
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-343, 346-349, 351-354, 356-357, 359-380, 382-386, 388-393, 396-410, 412-426, 428-435, 437-442, 444-449, 451-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 99
	v: [456-457, 460-461, 463-464, 466, 468, 470-475, 477-499, 502, 504-505, 511-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-577, 579-586, 588-590, 593-602, 604-616, 618-621, 623-625, 627-637, 639-647, 649-654, 656-662, 664-670, 672-682]


2021/12/16 05:55:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 05:55:41 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/16 05:55:41 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/16 05:55:46 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/16 05:55:46 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/16 05:55:46 eventingSetup.go:46: No eventing-con process running
2021/12/16 05:55:46 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/16 05:55:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:55:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:55:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:55:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:55:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 05:55:59 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 05:56:04 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (267.26s)
=== RUN   TestN1QLRecursion
2021/12/16 05:56:07 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 05:56:08 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 05:56:08 Removing node(s): [] from the cluster
2021/12/16 05:56:13 Rebalance progress: 100
2021/12/16 05:56:19 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:56:25 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:56:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:56:36 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:56:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 05:56:37 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:56:42 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:56:47 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:56:52 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:56:57 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:57:02 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:57:07 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/16 05:57:07 App: TestN1QLRecursion_function1 got deployed
2021/12/16 05:57:07 TestN1QLRecursion_function1 is deployed
2021/12/16 05:57:07 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:57:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 05:57:08 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:57:13 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 05:57:18 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 05:57:23 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 05:57:28 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/16 05:57:28 App: TestN1QLRecursion_function3 got deployed
2021/12/16 05:57:28 TestN1QLRecursion_function3 is deployed
2021/12/16 05:57:28 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:57:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/16 05:57:28 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/16 05:57:28 Success: TestN1QLRecursion_function2 is not deployed
2021/12/16 05:57:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:57:34 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 05:57:39 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 05:57:44 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 05:57:49 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/16 05:57:49 App: TestN1QLRecursion_function1 got un-deployed
2021/12/16 05:57:49 No eventing-con process running
2021/12/16 05:57:49 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/16 05:57:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:57:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:57:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:58:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:58:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 05:58:07 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 05:58:13 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 05:58:18 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 05:58:23 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/16 05:58:23 App: TestN1QLRecursion_function3 got un-deployed
2021/12/16 05:58:23 No eventing-con process running
2021/12/16 05:58:23 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/16 05:58:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:58:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:58:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:58:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 05:58:38 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 05:58:43 Rebalance progress: 100
--- PASS: TestN1QLRecursion (155.47s)
=== RUN   TestN1QLAllowRecursion
2021/12/16 05:58:43 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 05:58:43 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 05:58:43 Removing node(s): [] from the cluster
2021/12/16 05:58:48 Rebalance progress: 100
2021/12/16 05:58:54 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:58:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:59:05 Context: Bucket flush, response code: 200 dump: 
2021/12/16 05:59:10 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:59:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/16 05:59:11 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:59:16 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 05:59:21 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 05:59:26 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 05:59:31 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/16 05:59:31 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/16 05:59:31 TestN1QLAllowRecursion_function1 is deployed
2021/12/16 05:59:31 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:59:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/16 05:59:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:59:38 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 05:59:43 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 05:59:48 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 05:59:53 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/16 05:59:53 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/16 05:59:53 TestN1QLAllowRecursion_function3 is deployed
2021/12/16 05:59:53 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/16 05:59:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/16 05:59:54 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/16 05:59:59 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 06:00:04 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 06:00:09 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 06:00:14 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/16 06:00:14 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/16 06:00:14 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/16 06:00:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:00:19 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 06:00:24 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 06:00:29 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 06:00:34 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/16 06:00:34 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/16 06:00:34 No eventing-con process running
2021/12/16 06:00:34 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/16 06:00:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:00:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:00:43 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:00:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:00:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:00:53 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 06:00:58 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 06:01:03 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 06:01:08 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/16 06:01:08 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/16 06:01:08 No eventing-con process running
2021/12/16 06:01:08 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/16 06:01:10 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:01:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:01:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:01:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:01:27 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 06:01:32 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 06:01:37 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 06:01:42 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/16 06:01:42 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/16 06:01:42 No eventing-con process running
2021/12/16 06:01:42 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/16 06:01:44 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:01:51 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:01:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:01:57 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 06:02:02 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (199.57s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/16 06:02:02 Sleeping for some time
2021/12/16 06:02:12 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 06:02:12 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 06:02:12 No eventing-con process running
2021/12/16 06:02:12 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/16 06:02:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:02:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:02:21 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:02:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:02:26 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:02:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 06:02:27 Starting bucket ops to source bucket
2021/12/16 06:03:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:03:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:03:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:03:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 06:03:26 Starting bucket ops to source bucket
2021/12/16 06:03:28 Starting bucket ops to source bucket
2021/12/16 06:04:33 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/16 06:04:38 Waiting for dst bucket item count to get to: 7000 curr count: 6000
2021/12/16 06:04:38 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/16 06:04:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:04:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:04:43 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 06:04:48 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 06:04:53 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 06:04:58 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/16 06:04:58 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/16 06:04:58 No eventing-con process running
2021/12/16 06:04:59 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/16 06:05:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:05:10 Waiting for dst bucket item count to get to: 0 curr count: 740
2021/12/16 06:05:15 Waiting for dst bucket item count to get to: 0 curr count: 740
2021/12/16 06:05:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:05:17 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:05:27 Waiting for dst bucket item count to get to: 0 curr count: 7000
2021/12/16 06:05:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.99s)
=== RUN   TestTimerOverWriteSameReference
2021/12/16 06:05:32 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 06:05:32 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 06:05:32 No eventing-con process running
2021/12/16 06:05:32 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/16 06:05:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:05:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:05:41 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:05:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:05:46 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:05:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/16 06:05:52 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 06:05:57 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 06:06:02 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 06:06:07 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/16 06:06:07 App: TestTimerOverWriteSameReference got deployed
2021/12/16 06:06:17 Starting bucket ops to source bucket
2021/12/16 06:07:18 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/16 06:07:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:07:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:07:33 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 06:07:38 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 06:07:43 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 06:07:48 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/16 06:07:48 App: TestTimerOverWriteSameReference got un-deployed
2021/12/16 06:07:48 No eventing-con process running
2021/12/16 06:07:49 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/16 06:07:50 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:08:00 Waiting for dst bucket item count to get to: 0 curr count: 738
2021/12/16 06:08:05 Waiting for dst bucket item count to get to: 0 curr count: 738
2021/12/16 06:08:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:08:07 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:08:17 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/16 06:08:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.99s)
=== RUN   TestCancelTimerBucketop
2021/12/16 06:08:22 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 06:08:22 App: TestCancelTimerBucketop got un-deployed
2021/12/16 06:08:22 No eventing-con process running
2021/12/16 06:08:22 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/16 06:08:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:08:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:08:31 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:08:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:08:36 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:08:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/16 06:08:42 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 06:08:47 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 06:08:52 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 06:08:57 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/16 06:08:57 App: TestCancelTimerBucketop got deployed
2021/12/16 06:08:57 Starting bucket ops to source bucket
2021/12/16 06:09:04 Starting bucket ops to source bucket
2021/12/16 06:10:06 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 06:10:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:10:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:10:11 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 06:10:16 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 06:10:21 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 06:10:26 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/16 06:10:26 App: TestCancelTimerBucketop got un-deployed
2021/12/16 06:10:26 No eventing-con process running
2021/12/16 06:10:26 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/16 06:10:28 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:10:38 Waiting for dst bucket item count to get to: 0 curr count: 747
2021/12/16 06:10:43 Waiting for dst bucket item count to get to: 0 curr count: 247
2021/12/16 06:10:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:10:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:10:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:10:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (157.99s)
=== RUN   TestJSExpiryDate
2021/12/16 06:11:00 Starting bucket ops to source bucket
2021/12/16 06:11:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 06:11:13 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/16 06:11:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/16 06:11:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:11:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:11:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:11:50 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 06:11:55 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 06:12:00 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 06:12:05 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/16 06:12:05 App: TestJSExpiryDate got un-deployed
2021/12/16 06:12:05 No eventing-con process running
2021/12/16 06:12:05 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/16 06:12:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:12:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:12:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.71s)
=== RUN   TestBinaryDoc
2021/12/16 06:12:24 Starting bucket ops to source bucket
2021/12/16 06:12:24 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:12:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/16 06:12:25 Starting bucket ops to source bucket
2021/12/16 06:12:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:12:25 Deleting source bucket docs now
2021/12/16 06:12:25 Starting bucket ops to source bucket
2021/12/16 06:13:02 Waiting for dst bucket item count to get to: 32 curr count: 22
2021/12/16 06:13:07 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/16 06:13:12 Waiting for dst bucket item count to get to: 32 curr count: 31
2021/12/16 06:13:12 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/16 06:13:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:13:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:13:17 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 06:13:22 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 06:13:27 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 06:13:32 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/16 06:13:32 App: TestBinaryDoc got un-deployed
2021/12/16 06:13:32 No eventing-con process running
2021/12/16 06:13:33 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/16 06:13:34 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:13:44 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/16 06:13:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:13:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:13:56 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/16 06:13:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.23s)
=== RUN   TestConstantBindings
2021/12/16 06:13:56 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:13:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/16 06:13:57 Starting bucket ops to source bucket
2021/12/16 06:14:03 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 06:14:08 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 06:14:13 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/16 06:14:18 Waiting for dst bucket item count to get to: 6000 curr count: 481
2021/12/16 06:14:23 Waiting for dst bucket item count to get to: 6000 curr count: 481
2021/12/16 06:14:23 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/16 06:14:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:14:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:14:28 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 06:14:33 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 06:14:38 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 06:14:43 Waiting for app: TestConstantBindings to get un-deployed
2021/12/16 06:14:43 App: TestConstantBindings got un-deployed
2021/12/16 06:14:43 No eventing-con process running
2021/12/16 06:14:43 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/16 06:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:14:55 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:14:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:14:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:15:07 Waiting for dst bucket item count to get to: 0 curr count: 6000
2021/12/16 06:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (70.69s)
=== RUN   TestCodeInsight
2021/12/16 06:15:12 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 06:15:12 App: TestCodeInsight got un-deployed
2021/12/16 06:15:12 No eventing-con process running
2021/12/16 06:15:12 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/16 06:15:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:15:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:15:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:15:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:15:25 Starting bucket ops to source bucket
2021/12/16 06:15:26 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:15:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/16 06:15:32 Waiting for app: TestCodeInsight to get deployed
2021/12/16 06:15:37 Waiting for app: TestCodeInsight to get deployed
2021/12/16 06:15:42 Waiting for app: TestCodeInsight to get deployed
2021/12/16 06:15:47 Waiting for app: TestCodeInsight to get deployed
2021/12/16 06:15:47 App: TestCodeInsight got deployed
2021/12/16 06:15:52 Waiting for dst bucket item count to get to: 1000 curr count: 267
2021/12/16 06:15:52 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 06:15:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:15:58 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 06:16:03 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 06:16:08 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 06:16:13 Waiting for app: TestCodeInsight to get un-deployed
2021/12/16 06:16:13 App: TestCodeInsight got un-deployed
2021/12/16 06:16:13 No eventing-con process running
2021/12/16 06:16:13 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/16 06:16:14 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:16:24 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:16:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:16:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:16:36 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:16:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (89.50s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12669.743s

Functional tests took 3 hours, 31 minutes and 13 seconds.

N1ql tests

2021/12/16 06:16:45 Started n1ql
2021/12/16 06:16:55 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/16 06:16:56 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/16 06:16:56 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 06:16:56 Removing node(s): [] from the cluster
2021/12/16 06:17:01 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/16 06:17:06 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/16 06:17:06 App: TestRecursiveMutationN1QL got un-deployed
2021/12/16 06:17:06 No eventing-con process running
2021/12/16 06:17:06 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/16 06:17:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:17:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:17:15 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:17:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:17:20 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:17:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.61s)
=== RUN   TestFlexReset1
2021/12/16 06:17:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 06:17:25 App: TestFlexReset1 got un-deployed
2021/12/16 06:17:25 No eventing-con process running
2021/12/16 06:17:25 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/16 06:17:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:17:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:17:33 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:17:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:17:38 Starting bucket ops to source bucket
2021/12/16 06:17:39 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:17:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/16 06:17:45 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 06:17:50 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 06:17:55 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 06:18:00 Waiting for app: TestFlexReset1 to get deployed
2021/12/16 06:18:00 App: TestFlexReset1 got deployed
2021/12/16 06:18:05 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/16 06:18:05 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 06:18:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:18:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:18:10 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 06:18:15 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 06:18:20 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 06:18:25 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/16 06:18:25 App: TestFlexReset1 got un-deployed
2021/12/16 06:18:25 No eventing-con process running
2021/12/16 06:18:25 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/16 06:18:27 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:18:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:18:39 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:18:49 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:18:54 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 06:18:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.22s)
=== RUN   TestFlexReset2
2021/12/16 06:18:59 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 06:18:59 App: TestFlexReset2 got un-deployed
2021/12/16 06:18:59 No eventing-con process running
2021/12/16 06:18:59 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/16 06:19:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:19:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:19:08 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:19:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:19:13 Starting bucket ops to source bucket
2021/12/16 06:19:13 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:19:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/16 06:19:19 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 06:19:24 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 06:19:29 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 06:19:34 Waiting for app: TestFlexReset2 to get deployed
2021/12/16 06:19:34 App: TestFlexReset2 got deployed
2021/12/16 06:19:39 Waiting for dst bucket item count to get to: 1000 curr count: 545
2021/12/16 06:19:39 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/16 06:19:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:19:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:19:45 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 06:19:50 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 06:19:55 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 06:20:00 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/16 06:20:00 App: TestFlexReset2 got un-deployed
2021/12/16 06:20:00 No eventing-con process running
2021/12/16 06:20:00 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/16 06:20:01 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:20:11 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:20:16 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/16 06:20:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:20:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:20:28 Waiting for dst bucket item count to get to: 0 curr count: 749
2021/12/16 06:20:33 Waiting for dst bucket item count to get to: 0 curr count: 249
2021/12/16 06:20:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (99.22s)
=== RUN   TestN1QLLabelledBreak
2021/12/16 06:20:38 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 06:20:38 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 06:20:38 No eventing-con process running
2021/12/16 06:20:38 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/16 06:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:20:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:20:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:20:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:20:52 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:20:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 06:20:58 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 06:21:03 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 06:21:08 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 06:21:13 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/16 06:21:13 App: TestN1QLLabelledBreak got deployed
2021/12/16 06:21:13 Starting bucket ops to source bucket
2021/12/16 06:21:23 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 06:21:28 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 06:21:33 Waiting for dst bucket item count to get to: 10000 curr count: 932
2021/12/16 06:21:38 Waiting for dst bucket item count to get to: 10000 curr count: 2905
2021/12/16 06:21:43 Waiting for dst bucket item count to get to: 10000 curr count: 4975
2021/12/16 06:21:48 Waiting for dst bucket item count to get to: 10000 curr count: 7017
2021/12/16 06:21:53 Waiting for dst bucket item count to get to: 10000 curr count: 9022
2021/12/16 06:21:58 Waiting for dst bucket item count to get to: 10000 curr count: 9718
2021/12/16 06:21:58 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 06:21:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:21:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:22:03 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 06:22:08 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 06:22:13 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 06:22:18 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/16 06:22:18 App: TestN1QLLabelledBreak got un-deployed
2021/12/16 06:22:18 No eventing-con process running
2021/12/16 06:22:18 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/16 06:22:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:22:30 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 06:22:35 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 06:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:22:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:22:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 06:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (133.31s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/16 06:22:51 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 06:22:51 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 06:22:52 No eventing-con process running
2021/12/16 06:22:52 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/16 06:22:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:22:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:23:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:23:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:23:05 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:23:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/16 06:23:11 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 06:23:16 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 06:23:21 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 06:23:26 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/16 06:23:26 App: TestN1QLUnlabelledBreak got deployed
2021/12/16 06:23:26 Starting bucket ops to source bucket
2021/12/16 06:23:36 Waiting for dst bucket item count to get to: 10000 curr count: 371
2021/12/16 06:23:41 Waiting for dst bucket item count to get to: 10000 curr count: 371
2021/12/16 06:23:46 Waiting for dst bucket item count to get to: 10000 curr count: 5107
2021/12/16 06:23:51 Waiting for dst bucket item count to get to: 10000 curr count: 5107
2021/12/16 06:23:56 Waiting for dst bucket item count to get to: 10000 curr count: 5107
2021/12/16 06:23:56 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 06:23:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:23:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:24:01 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 06:24:06 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 06:24:11 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 06:24:16 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/16 06:24:16 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/16 06:24:16 No eventing-con process running
2021/12/16 06:24:16 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/16 06:24:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:24:28 Waiting for dst bucket item count to get to: 0 curr count: 3807
2021/12/16 06:24:33 Waiting for dst bucket item count to get to: 0 curr count: 1307
2021/12/16 06:24:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:24:35 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:24:45 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/16 06:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.48s)
=== RUN   TestN1QLThrowStatement
2021/12/16 06:24:50 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 06:24:50 App: TestN1QLThrowStatement got un-deployed
2021/12/16 06:24:50 No eventing-con process running
2021/12/16 06:24:50 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/16 06:24:52 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:24:59 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:25:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:25:04 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:25:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/16 06:25:10 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 06:25:15 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 06:25:20 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 06:25:25 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/16 06:25:25 App: TestN1QLThrowStatement got deployed
2021/12/16 06:25:25 Starting bucket ops to source bucket
2021/12/16 06:25:34 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/16 06:25:39 Waiting for dst bucket item count to get to: 10000 curr count: 166
2021/12/16 06:25:44 Waiting for dst bucket item count to get to: 10000 curr count: 534
2021/12/16 06:25:49 Waiting for dst bucket item count to get to: 10000 curr count: 2327
2021/12/16 06:25:54 Waiting for dst bucket item count to get to: 10000 curr count: 4363
2021/12/16 06:25:59 Waiting for dst bucket item count to get to: 10000 curr count: 8374
2021/12/16 06:26:04 Waiting for dst bucket item count to get to: 10000 curr count: 8374
2021/12/16 06:26:04 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/16 06:26:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:26:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:26:09 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 06:26:14 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 06:26:19 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 06:26:24 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/16 06:26:24 App: TestN1QLThrowStatement got un-deployed
2021/12/16 06:26:24 No eventing-con process running
2021/12/16 06:26:24 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/16 06:26:26 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:26:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 06:26:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:26:38 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:26:48 Waiting for dst bucket item count to get to: 0 curr count: 6430
2021/12/16 06:26:53 Waiting for dst bucket item count to get to: 0 curr count: 1430
2021/12/16 06:26:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (127.86s)
=== RUN   TestN1QLNestedForLoop
2021/12/16 06:26:58 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 06:26:58 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 06:26:58 No eventing-con process running
2021/12/16 06:26:58 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/16 06:27:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:27:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:27:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:27:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:27:11 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:27:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/16 06:27:18 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 06:27:23 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 06:27:28 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 06:27:33 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/16 06:27:33 App: TestN1QLNestedForLoop got deployed
2021/12/16 06:27:33 Starting bucket ops to source bucket
2021/12/16 06:27:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:27:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:27:52 Waiting for dst bucket item count to get to: 5000 curr count: 595
2021/12/16 06:27:57 Waiting for dst bucket item count to get to: 5000 curr count: 595
2021/12/16 06:28:02 Waiting for dst bucket item count to get to: 5000 curr count: 2970
2021/12/16 06:28:07 Waiting for dst bucket item count to get to: 5000 curr count: 2970
2021/12/16 06:28:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 06:28:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:28:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:28:12 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 06:28:17 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 06:28:22 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 06:28:27 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/16 06:28:27 App: TestN1QLNestedForLoop got un-deployed
2021/12/16 06:28:27 No eventing-con process running
2021/12/16 06:28:27 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/16 06:28:29 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:28:39 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 06:28:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/16 06:28:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:28:46 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:28:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 06:28:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (122.99s)
=== RUN   TestN1QLAutoClose
2021/12/16 06:29:01 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 06:29:01 App: TestN1QLAutoClose got un-deployed
2021/12/16 06:29:01 No eventing-con process running
2021/12/16 06:29:01 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/16 06:29:03 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:29:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:29:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:29:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:29:14 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:29:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/16 06:29:21 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 06:29:26 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 06:29:31 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 06:29:36 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/16 06:29:36 App: TestN1QLAutoClose got deployed
2021/12/16 06:29:36 Starting bucket ops to source bucket
2021/12/16 06:29:41 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/16 06:29:46 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/16 06:29:51 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/16 06:29:56 Waiting for dst bucket item count to get to: 500 curr count: 27
2021/12/16 06:30:01 Waiting for dst bucket item count to get to: 500 curr count: 27
2021/12/16 06:30:06 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/16 06:30:11 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/16 06:30:16 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/16 06:30:21 Waiting for dst bucket item count to get to: 500 curr count: 68
2021/12/16 06:30:26 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 06:30:31 Waiting for dst bucket item count to get to: 500 curr count: 89
2021/12/16 06:30:36 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/16 06:30:41 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/16 06:30:46 Waiting for dst bucket item count to get to: 500 curr count: 110
2021/12/16 06:30:51 Waiting for dst bucket item count to get to: 500 curr count: 130
2021/12/16 06:30:56 Waiting for dst bucket item count to get to: 500 curr count: 130
2021/12/16 06:31:01 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/16 06:31:06 Waiting for dst bucket item count to get to: 500 curr count: 153
2021/12/16 06:31:11 Waiting for dst bucket item count to get to: 500 curr count: 174
2021/12/16 06:31:16 Waiting for dst bucket item count to get to: 500 curr count: 174
2021/12/16 06:31:21 Waiting for dst bucket item count to get to: 500 curr count: 196
2021/12/16 06:31:26 Waiting for dst bucket item count to get to: 500 curr count: 196
2021/12/16 06:31:31 Waiting for dst bucket item count to get to: 500 curr count: 216
2021/12/16 06:31:36 Waiting for dst bucket item count to get to: 500 curr count: 216
2021/12/16 06:31:42 Waiting for dst bucket item count to get to: 500 curr count: 238
2021/12/16 06:31:47 Waiting for dst bucket item count to get to: 500 curr count: 246
2021/12/16 06:31:52 Waiting for dst bucket item count to get to: 500 curr count: 258
2021/12/16 06:31:57 Waiting for dst bucket item count to get to: 500 curr count: 267
2021/12/16 06:32:02 Waiting for dst bucket item count to get to: 500 curr count: 278
2021/12/16 06:32:07 Waiting for dst bucket item count to get to: 500 curr count: 291
2021/12/16 06:32:12 Waiting for dst bucket item count to get to: 500 curr count: 300
2021/12/16 06:32:17 Waiting for dst bucket item count to get to: 500 curr count: 312
2021/12/16 06:32:22 Waiting for dst bucket item count to get to: 500 curr count: 321
2021/12/16 06:32:27 Waiting for dst bucket item count to get to: 500 curr count: 332
2021/12/16 06:32:32 Waiting for dst bucket item count to get to: 500 curr count: 341
2021/12/16 06:32:37 Waiting for dst bucket item count to get to: 500 curr count: 353
2021/12/16 06:32:42 Waiting for dst bucket item count to get to: 500 curr count: 361
2021/12/16 06:32:47 Waiting for dst bucket item count to get to: 500 curr count: 372
2021/12/16 06:32:52 Waiting for dst bucket item count to get to: 500 curr count: 383
2021/12/16 06:32:57 Waiting for dst bucket item count to get to: 500 curr count: 392
2021/12/16 06:33:02 Waiting for dst bucket item count to get to: 500 curr count: 402
2021/12/16 06:33:07 Waiting for dst bucket item count to get to: 500 curr count: 414
2021/12/16 06:33:12 Waiting for dst bucket item count to get to: 500 curr count: 424
2021/12/16 06:33:17 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/16 06:33:22 Waiting for dst bucket item count to get to: 500 curr count: 447
2021/12/16 06:33:27 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/16 06:33:32 Waiting for dst bucket item count to get to: 500 curr count: 468
2021/12/16 06:33:37 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/16 06:33:42 Waiting for dst bucket item count to get to: 500 curr count: 489
2021/12/16 06:33:42 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/16 06:33:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:33:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:33:47 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 06:33:52 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 06:33:57 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 06:34:02 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/16 06:34:02 App: TestN1QLAutoClose got un-deployed
2021/12/16 06:34:02 No eventing-con process running
2021/12/16 06:34:02 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/16 06:34:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:34:14 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/16 06:34:19 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/16 06:34:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:34:20 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:34:30 Waiting for dst bucket item count to get to: 0 curr count: 403
2021/12/16 06:34:35 Waiting for dst bucket item count to get to: 0 curr count: 403
2021/12/16 06:34:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.67s)
=== RUN   TestN1QLPosParams
2021/12/16 06:34:40 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 06:34:40 App: TestN1QLPosParams got un-deployed
2021/12/16 06:34:40 No eventing-con process running
2021/12/16 06:34:41 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/16 06:34:42 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:34:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:34:49 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:34:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:34:54 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:34:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/16 06:35:00 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 06:35:05 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 06:35:10 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 06:35:15 Waiting for app: TestN1QLPosParams to get deployed
2021/12/16 06:35:15 App: TestN1QLPosParams got deployed
2021/12/16 06:35:15 Starting bucket ops to source bucket
2021/12/16 06:35:25 Waiting for dst bucket item count to get to: 5000 curr count: 981
2021/12/16 06:35:30 Waiting for dst bucket item count to get to: 5000 curr count: 981
2021/12/16 06:35:35 Waiting for dst bucket item count to get to: 5000 curr count: 2812
2021/12/16 06:35:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 06:35:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:35:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:35:40 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 06:35:45 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 06:35:50 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 06:35:55 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/16 06:35:55 App: TestN1QLPosParams got un-deployed
2021/12/16 06:35:55 No eventing-con process running
2021/12/16 06:35:55 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/16 06:35:57 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:36:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 06:36:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:36:09 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:36:19 Waiting for dst bucket item count to get to: 0 curr count: 4530
2021/12/16 06:36:24 Waiting for dst bucket item count to get to: 0 curr count: 2030
2021/12/16 06:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (108.20s)
=== RUN   TestN1QLExhaustConnPool
2021/12/16 06:36:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 06:36:29 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 06:36:29 No eventing-con process running
2021/12/16 06:36:29 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/16 06:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:36:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:36:37 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:36:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:36:42 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:36:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/16 06:36:48 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 06:36:53 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 06:36:58 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 06:37:03 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/16 06:37:03 App: TestN1QLExhaustConnPool got deployed
2021/12/16 06:37:03 Starting bucket ops to source bucket
2021/12/16 06:37:09 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/16 06:37:14 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/16 06:37:14 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/16 06:37:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:37:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:37:19 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 06:37:24 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 06:37:29 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 06:37:34 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/16 06:37:34 App: TestN1QLExhaustConnPool got un-deployed
2021/12/16 06:37:34 No eventing-con process running
2021/12/16 06:37:34 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/16 06:37:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:37:46 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/16 06:37:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:37:47 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:37:57 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/16 06:37:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.70s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/16 06:38:02 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 06:38:02 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 06:38:02 No eventing-con process running
2021/12/16 06:38:02 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/16 06:38:04 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:38:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:38:11 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:38:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:38:16 Starting bucket ops to source bucket
2021/12/16 06:38:33 Starting bucket ops to source bucket
2021/12/16 06:38:51 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 8271
2021/12/16 06:38:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/16 06:38:57 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 06:39:02 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 06:39:07 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 06:39:12 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/16 06:39:12 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/16 06:39:12 Starting bucket ops to source bucket
2021/12/16 06:39:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:32 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:52 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:39:57 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:40:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/16 06:40:02 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/16 06:40:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:40:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:40:07 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 06:40:12 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 06:40:17 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 06:40:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/16 06:40:22 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/16 06:40:22 No eventing-con process running
2021/12/16 06:40:23 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/16 06:40:24 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:40:34 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/16 06:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:40:36 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:40:46 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/16 06:40:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (168.82s)
=== RUN   TestN1QLQueryConsistency
2021/12/16 06:40:51 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 06:40:51 App: n1qlquery_consistency got un-deployed
2021/12/16 06:40:51 No eventing-con process running
2021/12/16 06:40:51 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/16 06:40:53 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:40:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:41:00 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:41:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:41:05 Starting bucket ops to source bucket
2021/12/16 06:41:08 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:41:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/16 06:41:14 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 06:41:19 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 06:41:24 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 06:41:29 Waiting for app: n1qlquery_consistency to get deployed
2021/12/16 06:41:29 App: n1qlquery_consistency got deployed
2021/12/16 06:41:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/16 06:41:39 Waiting for dst bucket item count to get to: 5000 curr count: 4334
2021/12/16 06:41:44 Waiting for dst bucket item count to get to: 5000 curr count: 4334
2021/12/16 06:41:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/16 06:41:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/16 06:41:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:41:49 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 06:41:54 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 06:41:59 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 06:42:04 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/16 06:42:04 App: n1qlquery_consistency got un-deployed
2021/12/16 06:42:04 No eventing-con process running
2021/12/16 06:42:04 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/16 06:42:06 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:42:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/16 06:42:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:42:18 Context: Bucket flush, response code: 200 dump: 
2021/12/16 06:42:28 Waiting for dst bucket item count to get to: 0 curr count: 3026
2021/12/16 06:42:33 Waiting for dst bucket item count to get to: 0 curr count: 526
2021/12/16 06:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.57s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1542.995s

N1ql tests took 0 hours, 25 minutes and 48 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/16 06:44:05 Started rebalance
Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:40876->127.0.0.1:9000: read: connection reset by peer
Quota setup Post "http://127.0.0.1:9000/pools/default": read tcp 127.0.0.1:40876->127.0.0.1:9000: read: connection reset by peer
=== RUN   TestDuplicateWithEventingRebIn
2021/12/16 06:44:24 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/16 06:44:29 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 06:44:29 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 06:44:29 No eventing-con process running
2021/12/16 06:44:29 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/16 06:44:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:44:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:44:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:44:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:44:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:44:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:44:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:44:47 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:44:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 06:44:53 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 06:44:58 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 06:45:03 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 06:45:08 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/16 06:45:08 App: TestDuplicateWithEventingRebIn got deployed
2021/12/16 06:45:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 06:45:08 Starting bucket ops to source bucket
2021/12/16 06:45:22 Removing node(s): [] from the cluster
2021/12/16 06:45:27 Rebalance progress: 0
2021/12/16 06:45:32 Rebalance progress: 0
2021/12/16 06:45:37 Rebalance progress: 25
2021/12/16 06:45:42 Rebalance progress: 25
2021/12/16 06:45:47 Rebalance progress: 25
2021/12/16 06:45:52 Rebalance progress: 25
2021/12/16 06:45:57 Rebalance progress: 100
2021/12/16 06:46:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:46:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 06:46:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:46:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 06:46:08 Rebalance progress: 25
2021/12/16 06:46:13 Rebalance progress: 25
2021/12/16 06:46:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:46:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:46:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:46:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 06:46:18 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:46:18 Waiting for src bucket item count to get to: 200000 curr count: 96325
2021/12/16 06:46:23 Waiting for src bucket item count to get to: 200000 curr count: 96325
2021/12/16 06:46:28 Waiting for src bucket item count to get to: 200000 curr count: 112989
2021/12/16 06:46:33 Waiting for src bucket item count to get to: 200000 curr count: 112989
2021/12/16 06:46:38 Waiting for src bucket item count to get to: 200000 curr count: 129589
2021/12/16 06:46:43 Waiting for src bucket item count to get to: 200000 curr count: 129589
2021/12/16 06:46:48 Waiting for src bucket item count to get to: 200000 curr count: 145892
2021/12/16 06:46:53 Waiting for src bucket item count to get to: 200000 curr count: 145892
2021/12/16 06:46:58 Waiting for src bucket item count to get to: 200000 curr count: 162339
2021/12/16 06:47:03 Waiting for src bucket item count to get to: 200000 curr count: 162339
2021/12/16 06:47:08 Waiting for src bucket item count to get to: 200000 curr count: 178725
2021/12/16 06:47:13 Waiting for src bucket item count to get to: 200000 curr count: 178725
2021/12/16 06:47:18 Waiting for src bucket item count to get to: 200000 curr count: 195305
2021/12/16 06:47:28 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:33 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:38 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:43 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:47:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:48:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:48:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:48:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/16 06:48:13 Post SET ops. Expected item count: 200000 got 200001
2021/12/16 06:48:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:48:18 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 06:48:23 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 06:48:28 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 06:48:33 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/16 06:48:33 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/16 06:48:33 No eventing-con process running
2021/12/16 06:48:33 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/16 06:48:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:48:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:48:45 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 06:48:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:48:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:48:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:48:56 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/16 06:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (282.70s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/16 06:49:01 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 06:49:02 Removing node(s): [] from the cluster
2021/12/16 06:49:07 Rebalance progress: 100
2021/12/16 06:49:12 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 06:49:12 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 06:49:12 No eventing-con process running
2021/12/16 06:49:12 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/16 06:49:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:49:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:49:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:49:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:49:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:49:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:49:30 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:49:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 06:49:36 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 06:49:41 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 06:49:46 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 06:49:51 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/16 06:49:51 App: TestDuplicateWithEventingRebOut got deployed
2021/12/16 06:49:51 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 06:49:51 Starting bucket ops to source bucket
2021/12/16 06:49:56 Rebalance progress: 25
2021/12/16 06:50:01 Rebalance progress: 25
2021/12/16 06:50:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:50:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:50:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:50:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 06:50:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:50:06 Waiting for src bucket item count to get to: 200000 curr count: 12320
2021/12/16 06:50:11 Waiting for src bucket item count to get to: 200000 curr count: 28490
2021/12/16 06:50:16 Waiting for src bucket item count to get to: 200000 curr count: 28490
2021/12/16 06:50:21 Waiting for src bucket item count to get to: 200000 curr count: 44984
2021/12/16 06:50:26 Waiting for src bucket item count to get to: 200000 curr count: 44984
2021/12/16 06:50:31 Waiting for src bucket item count to get to: 200000 curr count: 61141
2021/12/16 06:50:36 Waiting for src bucket item count to get to: 200000 curr count: 61141
2021/12/16 06:50:41 Waiting for src bucket item count to get to: 200000 curr count: 61141
2021/12/16 06:50:46 Waiting for src bucket item count to get to: 200000 curr count: 76925
2021/12/16 06:50:51 Waiting for src bucket item count to get to: 200000 curr count: 76925
2021/12/16 06:50:56 Waiting for src bucket item count to get to: 200000 curr count: 93147
2021/12/16 06:51:02 Waiting for src bucket item count to get to: 200000 curr count: 93147
2021/12/16 06:51:07 Waiting for src bucket item count to get to: 200000 curr count: 109785
2021/12/16 06:51:12 Waiting for src bucket item count to get to: 200000 curr count: 109785
2021/12/16 06:51:17 Waiting for src bucket item count to get to: 200000 curr count: 126195
2021/12/16 06:51:22 Waiting for src bucket item count to get to: 200000 curr count: 126195
2021/12/16 06:51:27 Waiting for src bucket item count to get to: 200000 curr count: 142627
2021/12/16 06:51:32 Waiting for src bucket item count to get to: 200000 curr count: 142627
2021/12/16 06:51:37 Waiting for src bucket item count to get to: 200000 curr count: 159279
2021/12/16 06:51:42 Waiting for src bucket item count to get to: 200000 curr count: 159279
2021/12/16 06:51:47 Waiting for src bucket item count to get to: 200000 curr count: 175491
2021/12/16 06:51:52 Waiting for src bucket item count to get to: 200000 curr count: 175491
2021/12/16 06:51:57 Waiting for src bucket item count to get to: 200000 curr count: 191833
2021/12/16 06:52:02 Waiting for src bucket item count to get to: 200000 curr count: 191833
2021/12/16 06:52:07 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 06:52:07 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 06:52:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:52:12 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 06:52:17 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 06:52:22 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 06:52:27 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/16 06:52:27 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/16 06:52:27 No eventing-con process running
2021/12/16 06:52:27 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/16 06:52:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:52:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:52:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 06:52:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:52:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:52:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:52:50 Waiting for dst bucket item count to get to: 0 curr count: 62610
2021/12/16 06:52:55 Waiting for dst bucket item count to get to: 0 curr count: 62610
2021/12/16 06:52:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.89s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/16 06:53:05 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 06:53:05 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 06:53:05 No eventing-con process running
2021/12/16 06:53:05 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/16 06:53:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:53:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:53:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:53:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:53:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:53:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:53:24 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:53:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 06:53:30 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 06:53:35 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 06:53:40 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 06:53:45 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/16 06:53:45 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/16 06:53:45 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 06:53:45 Starting bucket ops to source bucket
2021/12/16 06:53:45 Removing node(s): [] from the cluster
2021/12/16 06:53:50 Rebalance progress: 25
2021/12/16 06:53:55 Rebalance progress: 25
2021/12/16 06:54:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:54:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/16 06:54:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:54:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 06:54:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 06:54:13 Rebalance progress: 16.66666666666667
2021/12/16 06:54:18 Rebalance progress: 16.66666666666667
2021/12/16 06:54:23 Rebalance progress: 16.66666666666667
2021/12/16 06:54:28 Rebalance progress: 72.0558882235529
2021/12/16 06:54:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:54:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:54:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 06:54:38 Rebalance progress: 16.66666666666667
2021/12/16 06:54:43 Rebalance progress: 16.66666666666667
2021/12/16 06:54:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:54:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:54:48 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:54:48 Waiting for src bucket item count to get to: 200000 curr count: 81473
2021/12/16 06:54:53 Waiting for src bucket item count to get to: 200000 curr count: 97473
2021/12/16 06:54:58 Waiting for src bucket item count to get to: 200000 curr count: 97473
2021/12/16 06:55:03 Waiting for src bucket item count to get to: 200000 curr count: 113840
2021/12/16 06:55:08 Waiting for src bucket item count to get to: 200000 curr count: 113840
2021/12/16 06:55:13 Waiting for src bucket item count to get to: 200000 curr count: 129797
2021/12/16 06:55:18 Waiting for src bucket item count to get to: 200000 curr count: 129797
2021/12/16 06:55:24 Waiting for src bucket item count to get to: 200000 curr count: 146083
2021/12/16 06:55:29 Waiting for src bucket item count to get to: 200000 curr count: 146083
2021/12/16 06:55:34 Waiting for src bucket item count to get to: 200000 curr count: 162159
2021/12/16 06:55:39 Waiting for src bucket item count to get to: 200000 curr count: 162159
2021/12/16 06:55:44 Waiting for src bucket item count to get to: 200000 curr count: 177523
2021/12/16 06:55:49 Waiting for src bucket item count to get to: 200000 curr count: 177523
2021/12/16 06:55:54 Waiting for src bucket item count to get to: 200000 curr count: 193821
2021/12/16 06:55:59 Waiting for src bucket item count to get to: 200000 curr count: 193821
2021/12/16 06:56:04 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 06:56:04 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 06:56:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 06:56:09 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 06:56:14 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 06:56:19 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 06:56:24 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/16 06:56:24 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/16 06:56:24 No eventing-con process running
2021/12/16 06:56:24 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/16 06:56:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:56:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:56:35 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 06:56:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:56:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:56:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:56:47 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 06:56:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (232.01s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/16 06:56:57 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 06:56:57 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 06:56:57 No eventing-con process running
2021/12/16 06:56:57 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/16 06:56:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:57:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:57:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 06:57:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:57:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:57:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:57:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 06:57:16 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 06:57:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 06:57:22 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 06:57:27 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 06:57:32 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 06:57:37 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/16 06:57:37 App: TestDuplicateWithKVRebIn got deployed
2021/12/16 06:57:37 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 06:57:37 Starting bucket ops to source bucket
2021/12/16 06:57:37 Removing node(s): [] from the cluster
2021/12/16 06:57:42 Rebalance progress: 16.66666666666667
2021/12/16 06:57:47 Rebalance progress: 16.66666666666667
2021/12/16 06:57:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:57:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:57:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:57:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:57:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 06:57:57 Rebalance progress: 16.66666666666667
2021/12/16 06:58:02 Rebalance progress: 16.66666666666667
2021/12/16 06:58:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:58:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 06:58:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 06:58:07 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 06:58:07 Waiting for src bucket item count to get to: 200000 curr count: 34740
2021/12/16 06:58:12 Waiting for src bucket item count to get to: 200000 curr count: 50925
2021/12/16 06:58:17 Waiting for src bucket item count to get to: 200000 curr count: 50925
2021/12/16 06:58:22 Waiting for src bucket item count to get to: 200000 curr count: 67245
2021/12/16 06:58:27 Waiting for src bucket item count to get to: 200000 curr count: 67245
2021/12/16 06:58:32 Waiting for src bucket item count to get to: 200000 curr count: 82571
2021/12/16 06:58:37 Waiting for src bucket item count to get to: 200000 curr count: 82571
2021/12/16 06:58:43 Waiting for src bucket item count to get to: 200000 curr count: 98759
2021/12/16 06:58:48 Waiting for src bucket item count to get to: 200000 curr count: 98759
2021/12/16 06:58:53 Waiting for src bucket item count to get to: 200000 curr count: 98759
2021/12/16 06:58:58 Waiting for src bucket item count to get to: 200000 curr count: 115255
2021/12/16 06:59:03 Waiting for src bucket item count to get to: 200000 curr count: 115255
2021/12/16 06:59:08 Waiting for src bucket item count to get to: 200000 curr count: 131629
2021/12/16 06:59:13 Waiting for src bucket item count to get to: 200000 curr count: 131629
2021/12/16 06:59:18 Waiting for src bucket item count to get to: 200000 curr count: 148212
2021/12/16 06:59:23 Waiting for src bucket item count to get to: 200000 curr count: 148212
2021/12/16 06:59:28 Waiting for src bucket item count to get to: 200000 curr count: 164777
2021/12/16 06:59:33 Waiting for src bucket item count to get to: 200000 curr count: 164777
2021/12/16 06:59:38 Waiting for src bucket item count to get to: 200000 curr count: 180395
2021/12/16 06:59:43 Waiting for src bucket item count to get to: 200000 curr count: 180395
2021/12/16 06:59:48 Waiting for src bucket item count to get to: 200000 curr count: 196789
2021/12/16 06:59:53 Waiting for src bucket item count to get to: 200000 curr count: 196789
2021/12/16 06:59:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 06:59:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 06:59:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:00:03 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 07:00:08 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 07:00:13 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 07:00:18 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/16 07:00:18 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/16 07:00:18 No eventing-con process running
2021/12/16 07:00:18 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/16 07:00:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:00:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:00:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:00:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:00:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:00:36 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 07:00:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (229.12s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/16 07:00:41 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 07:00:42 Removing node(s): [] from the cluster
2021/12/16 07:00:47 Rebalance progress: 100
2021/12/16 07:00:52 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 07:00:52 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 07:00:52 No eventing-con process running
2021/12/16 07:00:52 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/16 07:00:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:00:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:00:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:01:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:01:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:01:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:01:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:01:10 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:01:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 07:01:16 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 07:01:21 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 07:01:26 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 07:01:31 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/16 07:01:31 App: TestDuplicateWithKVRebOut got deployed
2021/12/16 07:01:31 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:01:31 Starting bucket ops to source bucket
2021/12/16 07:01:36 Rebalance progress: 16.66666666666667
2021/12/16 07:01:41 Rebalance progress: 16.66666666666667
2021/12/16 07:01:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:01:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:01:46 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:01:46 Waiting for src bucket item count to get to: 200000 curr count: 11831
2021/12/16 07:01:51 Waiting for src bucket item count to get to: 200000 curr count: 28111
2021/12/16 07:01:56 Waiting for src bucket item count to get to: 200000 curr count: 28111
2021/12/16 07:02:01 Waiting for src bucket item count to get to: 200000 curr count: 44271
2021/12/16 07:02:06 Waiting for src bucket item count to get to: 200000 curr count: 44271
2021/12/16 07:02:11 Waiting for src bucket item count to get to: 200000 curr count: 60397
2021/12/16 07:02:16 Waiting for src bucket item count to get to: 200000 curr count: 60397
2021/12/16 07:02:22 Waiting for src bucket item count to get to: 200000 curr count: 60397
2021/12/16 07:02:27 Waiting for src bucket item count to get to: 200000 curr count: 75908
2021/12/16 07:02:32 Waiting for src bucket item count to get to: 200000 curr count: 75908
2021/12/16 07:02:37 Waiting for src bucket item count to get to: 200000 curr count: 91917
2021/12/16 07:02:42 Waiting for src bucket item count to get to: 200000 curr count: 91917
2021/12/16 07:02:47 Waiting for src bucket item count to get to: 200000 curr count: 107914
2021/12/16 07:02:52 Waiting for src bucket item count to get to: 200000 curr count: 107914
2021/12/16 07:02:57 Waiting for src bucket item count to get to: 200000 curr count: 123931
2021/12/16 07:03:02 Waiting for src bucket item count to get to: 200000 curr count: 123931
2021/12/16 07:03:07 Waiting for src bucket item count to get to: 200000 curr count: 140124
2021/12/16 07:03:12 Waiting for src bucket item count to get to: 200000 curr count: 140124
2021/12/16 07:03:17 Waiting for src bucket item count to get to: 200000 curr count: 156534
2021/12/16 07:03:22 Waiting for src bucket item count to get to: 200000 curr count: 156534
2021/12/16 07:03:27 Waiting for src bucket item count to get to: 200000 curr count: 171993
2021/12/16 07:03:32 Waiting for src bucket item count to get to: 200000 curr count: 171993
2021/12/16 07:03:37 Waiting for src bucket item count to get to: 200000 curr count: 187853
2021/12/16 07:03:42 Waiting for src bucket item count to get to: 200000 curr count: 187853
2021/12/16 07:03:47 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 07:03:47 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 07:03:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:03:52 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 07:03:57 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 07:04:02 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 07:04:07 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/16 07:04:07 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/16 07:04:07 No eventing-con process running
2021/12/16 07:04:07 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/16 07:04:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:04:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:04:19 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 07:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:04:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:04:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:04:30 Waiting for dst bucket item count to get to: 0 curr count: 72206
2021/12/16 07:04:35 Waiting for dst bucket item count to get to: 0 curr count: 72206
2021/12/16 07:04:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (238.95s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/16 07:04:45 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 07:04:45 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 07:04:45 No eventing-con process running
2021/12/16 07:04:45 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/16 07:04:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:04:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:04:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:04:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:05:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:05:04 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:05:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 07:05:10 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 07:05:15 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 07:05:20 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 07:05:25 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/16 07:05:25 App: TestDuplicateWithKVSwapReb got deployed
2021/12/16 07:05:25 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 07:05:25 Starting bucket ops to source bucket
2021/12/16 07:05:26 Removing node(s): [] from the cluster
2021/12/16 07:05:31 Rebalance progress: 16.66666666666667
2021/12/16 07:05:36 Rebalance progress: 16.66666666666667
2021/12/16 07:05:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:05:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:05:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:05:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 07:05:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:05:46 Rebalance progress: 16.66666666666667
2021/12/16 07:05:51 Rebalance progress: 16.66666666666667
2021/12/16 07:05:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:05:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:05:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:05:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:06:01 Rebalance progress: 16.66666666666667
2021/12/16 07:06:06 Rebalance progress: 16.66666666666667
2021/12/16 07:06:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:06:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:06:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:06:11 Waiting for src bucket item count to get to: 200000 curr count: 70294
2021/12/16 07:06:16 Waiting for src bucket item count to get to: 200000 curr count: 70294
2021/12/16 07:06:21 Waiting for src bucket item count to get to: 200000 curr count: 85168
2021/12/16 07:06:26 Waiting for src bucket item count to get to: 200000 curr count: 85168
2021/12/16 07:06:31 Waiting for src bucket item count to get to: 200000 curr count: 101354
2021/12/16 07:06:36 Waiting for src bucket item count to get to: 200000 curr count: 101354
2021/12/16 07:06:41 Waiting for src bucket item count to get to: 200000 curr count: 117752
2021/12/16 07:06:46 Waiting for src bucket item count to get to: 200000 curr count: 117752
2021/12/16 07:06:51 Waiting for src bucket item count to get to: 200000 curr count: 133941
2021/12/16 07:06:56 Waiting for src bucket item count to get to: 200000 curr count: 133941
2021/12/16 07:07:01 Waiting for src bucket item count to get to: 200000 curr count: 150142
2021/12/16 07:07:06 Waiting for src bucket item count to get to: 200000 curr count: 150142
2021/12/16 07:07:11 Waiting for src bucket item count to get to: 200000 curr count: 166295
2021/12/16 07:07:16 Waiting for src bucket item count to get to: 200000 curr count: 166295
2021/12/16 07:07:21 Waiting for src bucket item count to get to: 200000 curr count: 181535
2021/12/16 07:07:26 Waiting for src bucket item count to get to: 200000 curr count: 181535
2021/12/16 07:07:31 Waiting for src bucket item count to get to: 200000 curr count: 197917
2021/12/16 07:07:36 Waiting for src bucket item count to get to: 200000 curr count: 197917
2021/12/16 07:07:41 Waiting for src bucket item count to get to: 200000 curr count: 197917
2021/12/16 07:07:46 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 07:07:46 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 07:07:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:07:52 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 07:07:57 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 07:08:02 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 07:08:07 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/16 07:08:07 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/16 07:08:07 No eventing-con process running
2021/12/16 07:08:07 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/16 07:08:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:08:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:08:18 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/16 07:08:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:08:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:08:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:08:30 Waiting for dst bucket item count to get to: 0 curr count: 60696
2021/12/16 07:08:35 Waiting for dst bucket item count to get to: 0 curr count: 60696
2021/12/16 07:08:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (239.96s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/16 07:08:45 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:08:45 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:08:45 No eventing-con process running
2021/12/16 07:08:45 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 07:08:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:08:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:08:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:08:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:08:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:08:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:09:04 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:09:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 07:09:10 Starting bucket ops to source bucket
2021/12/16 07:09:15 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:09:20 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:09:25 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:09:25 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:09:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:09:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:09:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:09:31 Rebalance progress: 16.66666666666667
2021/12/16 07:09:36 Rebalance progress: 16.66666666666667
2021/12/16 07:09:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:09:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:09:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:09:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:09:46 Rebalance progress: 16.66666666666667
2021/12/16 07:09:51 Rebalance progress: 16.66666666666667
2021/12/16 07:09:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:09:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:09:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/16 07:09:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 07:10:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:10:07 Rebalance progress: 12.5
2021/12/16 07:10:12 Rebalance progress: 12.5
2021/12/16 07:10:17 Rebalance progress: 12.5
2021/12/16 07:10:22 Rebalance progress: 62.9
2021/12/16 07:10:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:10:27 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:10:32 Rebalance progress: 12.5
2021/12/16 07:10:37 Rebalance progress: 12.5
2021/12/16 07:10:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:10:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:10:47 Rebalance progress: 12.5
2021/12/16 07:10:52 Rebalance progress: 12.5
2021/12/16 07:10:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:10:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:10:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 07:11:02 Rebalance progress: 12.5
2021/12/16 07:11:07 Rebalance progress: 12.5
2021/12/16 07:11:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:11:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:11:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:11:12 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:11:17 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:11:22 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:11:27 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:11:32 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:11:32 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:11:32 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:11:32 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 07:11:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:11:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:11:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12229
2021/12/16 07:11:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12229
2021/12/16 07:11:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:11:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:11:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:12:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11822
2021/12/16 07:12:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11822
2021/12/16 07:12:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (210.76s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/16 07:12:16 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 07:12:16 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 07:12:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:12:16 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/16 07:12:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:12:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:12:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:12:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:12:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:12:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:12:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:12:35 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:12:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/16 07:12:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:12:46 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 07:12:51 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 07:12:56 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/16 07:12:56 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:12:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:12:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:12:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:12:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:12:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 07:12:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:13:01 Rebalance progress: 12.5
2021/12/16 07:13:06 Rebalance progress: 12.5
2021/12/16 07:13:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:13:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:13:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:13:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 07:13:16 Rebalance progress: 12.5
2021/12/16 07:13:21 Rebalance progress: 12.5
2021/12/16 07:13:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:13:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:13:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:13:32 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 07:13:37 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 07:13:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 07:13:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/16 07:13:47 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/16 07:13:47 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:13:47 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/16 07:13:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:13:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4558
2021/12/16 07:13:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:14:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:14:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:14:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1333
2021/12/16 07:14:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1333
2021/12/16 07:14:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (129.24s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/16 07:14:25 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 07:14:25 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 07:14:25 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:14:25 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/16 07:14:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:14:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:14:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:14:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:14:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:14:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:14:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:14:44 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:14:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/16 07:14:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:14:55 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 07:15:00 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 07:15:05 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/16 07:15:05 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:15:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:15:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:15:10 Rebalance progress: 12.5
2021/12/16 07:15:15 Rebalance progress: 12.5
2021/12/16 07:15:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:15:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:15:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:15:26 Rebalance progress: 12.5
2021/12/16 07:15:31 Rebalance progress: 12.5
2021/12/16 07:15:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:15:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 07:15:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:15:41 Rebalance progress: 12.5
2021/12/16 07:15:46 Rebalance progress: 12.5
2021/12/16 07:15:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:15:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:15:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:15:56 Rebalance progress: 12.5
2021/12/16 07:16:01 Rebalance progress: 12.5
2021/12/16 07:16:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:16:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:16:11 Rebalance progress: 12.5
2021/12/16 07:16:16 Rebalance progress: 12.5
2021/12/16 07:16:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:16:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 07:16:26 Rebalance progress: 12.5
2021/12/16 07:16:31 Rebalance progress: 12.5
2021/12/16 07:16:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:16:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:16:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:16:42 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 07:16:47 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 07:16:52 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 07:16:57 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/16 07:16:57 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/16 07:16:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:16:57 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/16 07:16:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:17:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:17:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10647
2021/12/16 07:17:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:17:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:17:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:17:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3747
2021/12/16 07:17:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3747
2021/12/16 07:17:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (190.03s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/16 07:17:35 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 07:17:35 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 07:17:35 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:17:35 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/16 07:17:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:17:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:17:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:17:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:17:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:17:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:17:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:17:49 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:17:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 07:17:55 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:18:00 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 07:18:05 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 07:18:10 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/16 07:18:10 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:18:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:18:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:18:15 Rebalance progress: 12.5
2021/12/16 07:18:20 Rebalance progress: 12.5
2021/12/16 07:18:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:18:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:18:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:18:31 Rebalance progress: 12.5
2021/12/16 07:18:36 Rebalance progress: 12.5
2021/12/16 07:18:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:18:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:18:46 Rebalance progress: 12.5
2021/12/16 07:18:51 Rebalance progress: 12.5
2021/12/16 07:18:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:18:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:18:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:19:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 07:19:06 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 07:19:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 07:19:16 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/16 07:19:16 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/16 07:19:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:19:16 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/16 07:19:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:19:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:19:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6076
2021/12/16 07:19:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:19:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:19:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:19:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (129.38s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/16 07:19:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:19:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:19:45 Rebalance progress: 100
2021/12/16 07:19:50 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 07:19:50 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 07:19:50 No eventing-con process running
2021/12/16 07:19:50 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/16 07:19:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:19:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:19:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:19:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:20:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:20:04 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:20:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 07:20:10 Starting bucket ops to source bucket
2021/12/16 07:20:15 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 07:20:20 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 07:20:25 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/16 07:20:25 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:20:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:20:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:20:31 Rebalance progress: 12.5
2021/12/16 07:20:36 Rebalance progress: 12.5
2021/12/16 07:20:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:20:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 07:20:41 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 07:20:46 Rebalance progress: 12.5
2021/12/16 07:20:51 Rebalance progress: 12.5
2021/12/16 07:20:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:20:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:20:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:21:01 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 07:21:06 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 07:21:11 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 07:21:16 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/16 07:21:16 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/16 07:21:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:21:16 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/16 07:21:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:21:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:21:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4573
2021/12/16 07:21:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:21:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:21:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:21:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (119.82s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/16 07:21:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:21:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:21:45 Rebalance progress: 100
2021/12/16 07:21:50 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 07:21:50 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 07:21:50 No eventing-con process running
2021/12/16 07:21:50 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/16 07:21:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:21:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:21:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:21:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:22:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:22:03 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:22:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/16 07:22:09 Starting bucket ops to source bucket
2021/12/16 07:22:14 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 07:22:19 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 07:22:24 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/16 07:22:24 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:22:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:22:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:22:30 Rebalance progress: 12.5
2021/12/16 07:22:35 Rebalance progress: 12.5
2021/12/16 07:22:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:22:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 07:22:40 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 07:22:45 Rebalance progress: 12.5
2021/12/16 07:22:50 Rebalance progress: 12.5
2021/12/16 07:22:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:22:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:22:55 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:22:55 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/16 07:23:00 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 07:23:05 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 07:23:10 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 07:23:15 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/16 07:23:15 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/16 07:23:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:23:15 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/16 07:23:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:23:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:23:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37652
2021/12/16 07:23:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:23:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:23:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:23:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37652
2021/12/16 07:23:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (124.36s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/16 07:23:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 07:23:49 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 07:23:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:23:49 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/16 07:23:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:23:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:23:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:23:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:24:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:24:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:24:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:24:07 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:24:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:24:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:24:09 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:24:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 07:24:15 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:24:20 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 07:24:25 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 07:24:30 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/16 07:24:30 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/16 07:24:35 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 07:24:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 07:24:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/16 07:24:45 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:24:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:24:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:24:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:24:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:24:50 Rebalance progress: 12.5
2021/12/16 07:24:55 Rebalance progress: 12.5
2021/12/16 07:25:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:25:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:25:00 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:25:05 Rebalance progress: 12.5
2021/12/16 07:25:10 Rebalance progress: 12.5
2021/12/16 07:25:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:25:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 07:25:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:25:21 Rebalance progress: 12.5
2021/12/16 07:25:26 Rebalance progress: 12.5
2021/12/16 07:25:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:25:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:25:36 Rebalance progress: 12.5
2021/12/16 07:25:41 Rebalance progress: 12.5
2021/12/16 07:25:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:25:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:25:51 Rebalance progress: 12.5
2021/12/16 07:25:56 Rebalance progress: 12.5
2021/12/16 07:26:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:26:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 07:26:07 Rebalance progress: 12.5
2021/12/16 07:26:12 Rebalance progress: 12.5
2021/12/16 07:26:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:26:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:26:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 07:26:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 07:26:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 07:26:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/16 07:26:37 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/16 07:26:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:26:37 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/16 07:26:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:26:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:26:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:26:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:26:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12178
2021/12/16 07:26:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:26:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:27:02 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 07:27:07 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 07:27:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 07:27:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/16 07:27:17 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/16 07:27:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:27:17 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/16 07:27:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:27:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:27:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:27:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:27:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:27:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (231.92s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/16 07:27:41 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 07:27:41 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 07:27:41 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:27:41 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/16 07:27:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:27:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:27:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:27:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:27:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:27:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:27:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:27:59 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:28:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:28:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:28:01 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:28:02 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 07:28:07 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:28:12 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 07:28:17 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 07:28:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/16 07:28:22 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/16 07:28:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 07:28:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 07:28:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/16 07:28:37 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:28:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:28:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:28:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:28:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:28:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/16 07:28:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:28:43 Rebalance progress: 12.5
2021/12/16 07:28:48 Rebalance progress: 12.5
2021/12/16 07:28:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:28:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:28:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:28:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 07:28:58 Rebalance progress: 12.5
2021/12/16 07:29:03 Rebalance progress: 12.5
2021/12/16 07:29:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:29:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:29:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:29:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:29:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 07:29:18 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 07:29:23 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 07:29:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/16 07:29:28 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/16 07:29:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:29:28 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/16 07:29:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:29:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:29:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2527
2021/12/16 07:29:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2527
2021/12/16 07:29:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:29:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:29:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:29:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6083
2021/12/16 07:29:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:29:59 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:30:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 07:30:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 07:30:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 07:30:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/16 07:30:19 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/16 07:30:19 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:30:19 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/16 07:30:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:30:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:30:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:30:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:30:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.88s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/16 07:30:43 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 07:30:43 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 07:30:43 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:30:43 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/16 07:30:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:30:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:30:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:30:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:30:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:30:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:31:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 07:31:01 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 07:31:01 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:31:01 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/16 07:31:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:31:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:31:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:31:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:31:15 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:31:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:31:16 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:31:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 07:31:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:31:27 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 07:31:32 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 07:31:37 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/16 07:31:37 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/16 07:31:42 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 07:31:47 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 07:31:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/16 07:31:52 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:31:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:31:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:31:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:31:57 Rebalance progress: 12.5
2021/12/16 07:32:02 Rebalance progress: 12.5
2021/12/16 07:32:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:32:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:32:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:32:14 Rebalance progress: 12.5
2021/12/16 07:32:19 Rebalance progress: 12.5
2021/12/16 07:32:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:32:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:32:29 Rebalance progress: 12.5
2021/12/16 07:32:34 Rebalance progress: 12.5
2021/12/16 07:32:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:32:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:32:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:32:44 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 07:32:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 07:32:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 07:32:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/16 07:32:59 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/16 07:32:59 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:32:59 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/16 07:33:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:33:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:33:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6314
2021/12/16 07:33:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6314
2021/12/16 07:33:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:33:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:33:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:33:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7656
2021/12/16 07:33:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:33:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:33:34 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 07:33:39 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 07:33:44 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 07:33:49 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/16 07:33:49 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/16 07:33:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:33:49 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/16 07:33:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:33:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:33:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:33:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:34:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.06s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 07:34:13 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:34:13 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:34:13 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:34:13 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 07:34:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:34:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:34:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:34:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:34:31 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:34:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 07:34:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:34:42 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:34:47 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:34:52 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:34:52 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:34:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:34:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:34:53 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:35:13 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:35:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:35:49 Rebalance progress: 12.5
2021/12/16 07:35:54 Rebalance progress: 12.5
2021/12/16 07:35:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:35:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:35:59 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:35:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:36:04 Rebalance progress: 12.5
2021/12/16 07:36:09 Rebalance progress: 12.5
2021/12/16 07:36:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:36:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:36:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:36:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:36:19 Rebalance progress: 12.5
2021/12/16 07:36:24 Rebalance progress: 12.5
2021/12/16 07:36:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:36:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:36:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:36:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:36:34 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:36:39 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:36:44 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:36:49 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:36:49 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:36:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:36:49 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 07:36:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:36:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:37:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11161
2021/12/16 07:37:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11161
2021/12/16 07:37:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:37:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:37:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:37:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11161
2021/12/16 07:37:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (195.18s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 07:37:28 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:37:28 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:37:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:37:28 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/16 07:37:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:37:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:37:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:37:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:37:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:37:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:37:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:37:46 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:37:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/16 07:37:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:37:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:38:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:38:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/16 07:38:08 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:38:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:38:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:38:28 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:38:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:38:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:38:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:38:59 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:39:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:39:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:39:30 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:39:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:39:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:39:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:40:01 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:40:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:40:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:40:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:40:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:40:48 Rebalance progress: 12.5
2021/12/16 07:40:53 Rebalance progress: 12.5
2021/12/16 07:40:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:40:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:40:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:40:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:41:03 Rebalance progress: 12.5
2021/12/16 07:41:08 Rebalance progress: 12.5
2021/12/16 07:41:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:41:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:41:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:41:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:41:18 Rebalance progress: 12.5
2021/12/16 07:41:23 Rebalance progress: 12.5
2021/12/16 07:41:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:41:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:41:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:41:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:41:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:41:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:41:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:41:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/16 07:41:49 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/16 07:41:49 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:41:49 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/16 07:41:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:41:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:42:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12486
2021/12/16 07:42:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12486
2021/12/16 07:42:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:42:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:42:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:42:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21570
2021/12/16 07:42:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (299.54s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/16 07:42:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 07:42:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 07:42:27 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:42:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/16 07:42:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:42:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:42:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:42:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:42:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:42:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:42:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:42:46 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:42:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 07:42:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:42:57 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 07:43:02 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 07:43:07 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/16 07:43:07 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:43:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:43:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:43:12 Rebalance progress: 12.5
2021/12/16 07:43:17 Rebalance progress: 12.5
2021/12/16 07:43:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:43:22 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:43:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:43:38 Rebalance progress: 12.5
2021/12/16 07:43:43 Rebalance progress: 12.5
2021/12/16 07:43:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:43:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:43:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 07:43:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:43:53 Rebalance progress: 12.5
2021/12/16 07:43:58 Rebalance progress: 12.5
2021/12/16 07:44:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:44:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:44:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:44:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:44:08 Rebalance progress: 12.5
2021/12/16 07:44:13 Rebalance progress: 12.5
2021/12/16 07:44:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:44:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:44:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:44:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:44:23 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 07:44:28 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 07:44:33 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 07:44:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/16 07:44:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/16 07:44:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:44:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/16 07:44:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:44:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:44:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 405
2021/12/16 07:44:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 405
2021/12/16 07:44:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:44:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:45:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8575
2021/12/16 07:45:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.35s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/16 07:45:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 07:45:17 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 07:45:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:45:17 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/16 07:45:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:45:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:45:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:45:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:45:35 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:45:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/16 07:45:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:45:46 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 07:45:52 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 07:45:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/16 07:45:57 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:45:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:45:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/16 07:45:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:46:03 Rebalance progress: 12.5
2021/12/16 07:46:08 Rebalance progress: 12.5
2021/12/16 07:46:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:46:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/16 07:46:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:46:18 Rebalance progress: 12.5
2021/12/16 07:46:23 Rebalance progress: 12.5
2021/12/16 07:46:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:46:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 07:46:33 Rebalance progress: 12.5
2021/12/16 07:46:38 Rebalance progress: 12.5
2021/12/16 07:46:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:46:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:46:48 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 07:46:53 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 07:46:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 07:47:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/16 07:47:03 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/16 07:47:03 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:47:03 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/16 07:47:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:47:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:47:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6166
2021/12/16 07:47:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:47:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:47:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:47:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6166
2021/12/16 07:47:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (140.22s)
=== RUN   TestEventingRebInWithOps
2021/12/16 07:47:37 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 07:47:37 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/16 07:47:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 07:47:37 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/16 07:47:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:47:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:47:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:47:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:47:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:47:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:47:56 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:47:57 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 07:48:02 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 07:48:07 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 07:48:12 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 07:48:17 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/16 07:48:17 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/16 07:48:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:48:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 07:48:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 07:48:22 Rebalance progress: 12.5
2021/12/16 07:48:27 Rebalance progress: 12.5
2021/12/16 07:48:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:48:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:48:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:48:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:48:38 Rebalance progress: 12.5
2021/12/16 07:48:43 Rebalance progress: 12.5
2021/12/16 07:48:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:48:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:48:48 Waiting for src bucket item count to get to: 200000 curr count: 34138
2021/12/16 07:48:53 Waiting for src bucket item count to get to: 200000 curr count: 49894
2021/12/16 07:48:58 Waiting for src bucket item count to get to: 200000 curr count: 49894
2021/12/16 07:49:03 Waiting for src bucket item count to get to: 200000 curr count: 66051
2021/12/16 07:49:08 Waiting for src bucket item count to get to: 200000 curr count: 66051
2021/12/16 07:49:13 Waiting for src bucket item count to get to: 200000 curr count: 80883
2021/12/16 07:49:18 Waiting for src bucket item count to get to: 200000 curr count: 80883
2021/12/16 07:49:23 Waiting for src bucket item count to get to: 200000 curr count: 96919
2021/12/16 07:49:28 Waiting for src bucket item count to get to: 200000 curr count: 96919
2021/12/16 07:49:33 Waiting for src bucket item count to get to: 200000 curr count: 112818
2021/12/16 07:49:38 Waiting for src bucket item count to get to: 200000 curr count: 112818
2021/12/16 07:49:43 Waiting for src bucket item count to get to: 200000 curr count: 128929
2021/12/16 07:49:48 Waiting for src bucket item count to get to: 200000 curr count: 128929
2021/12/16 07:49:53 Waiting for src bucket item count to get to: 200000 curr count: 145052
2021/12/16 07:49:58 Waiting for src bucket item count to get to: 200000 curr count: 145052
2021/12/16 07:50:03 Waiting for src bucket item count to get to: 200000 curr count: 145052
2021/12/16 07:50:08 Waiting for src bucket item count to get to: 200000 curr count: 160864
2021/12/16 07:50:13 Waiting for src bucket item count to get to: 200000 curr count: 160864
2021/12/16 07:50:18 Waiting for src bucket item count to get to: 200000 curr count: 175859
2021/12/16 07:50:23 Waiting for src bucket item count to get to: 200000 curr count: 175859
2021/12/16 07:50:28 Waiting for src bucket item count to get to: 200000 curr count: 191894
2021/12/16 07:50:33 Waiting for src bucket item count to get to: 200000 curr count: 191894
2021/12/16 07:50:38 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 07:50:38 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 07:50:38 Starting bucket ops to source bucket
2021/12/16 07:54:28 Waiting for dst bucket item count to get to: 0 curr count: 6282
2021/12/16 07:54:33 Waiting for dst bucket item count to get to: 0 curr count: 6282
2021/12/16 07:54:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:54:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 07:54:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 07:54:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 07:54:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 07:54:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 07:54:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/16 07:54:53 App: TestEventingRebInWithOps got un-deployed
2021/12/16 07:54:53 No eventing-con process running
2021/12/16 07:54:53 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/16 07:54:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:55:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:55:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:55:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:55:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.56s)
=== RUN   TestEventingRebOutWithOps
2021/12/16 07:55:12 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 07:55:12 Removing node(s): [] from the cluster
2021/12/16 07:55:17 Rebalance progress: 100
2021/12/16 07:55:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 07:55:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 07:55:22 No eventing-con process running
2021/12/16 07:55:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/16 07:55:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:55:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:55:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 07:55:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:55:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:55:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:55:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 07:55:40 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 07:55:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 07:55:46 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 07:55:51 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 07:55:57 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 07:56:02 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/16 07:56:02 App: TestEventingRebOutWithOps got deployed
2021/12/16 07:56:02 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 07:56:02 Starting bucket ops to source bucket
2021/12/16 07:56:07 Rebalance progress: 12.5
2021/12/16 07:56:12 Rebalance progress: 12.5
2021/12/16 07:56:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 07:56:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 07:56:17 Waiting for src bucket item count to get to: 200000 curr count: 11394
2021/12/16 07:56:22 Waiting for src bucket item count to get to: 200000 curr count: 27197
2021/12/16 07:56:27 Waiting for src bucket item count to get to: 200000 curr count: 27197
2021/12/16 07:56:32 Waiting for src bucket item count to get to: 200000 curr count: 43297
2021/12/16 07:56:37 Waiting for src bucket item count to get to: 200000 curr count: 43297
2021/12/16 07:56:42 Waiting for src bucket item count to get to: 200000 curr count: 59302
2021/12/16 07:56:47 Waiting for src bucket item count to get to: 200000 curr count: 59302
2021/12/16 07:56:52 Waiting for src bucket item count to get to: 200000 curr count: 59302
2021/12/16 07:56:57 Waiting for src bucket item count to get to: 200000 curr count: 74721
2021/12/16 07:57:02 Waiting for src bucket item count to get to: 200000 curr count: 74721
2021/12/16 07:57:07 Waiting for src bucket item count to get to: 200000 curr count: 90343
2021/12/16 07:57:12 Waiting for src bucket item count to get to: 200000 curr count: 90343
2021/12/16 07:57:17 Waiting for src bucket item count to get to: 200000 curr count: 106399
2021/12/16 07:57:22 Waiting for src bucket item count to get to: 200000 curr count: 106399
2021/12/16 07:57:27 Waiting for src bucket item count to get to: 200000 curr count: 122570
2021/12/16 07:57:32 Waiting for src bucket item count to get to: 200000 curr count: 122570
2021/12/16 07:57:37 Waiting for src bucket item count to get to: 200000 curr count: 138670
2021/12/16 07:57:42 Waiting for src bucket item count to get to: 200000 curr count: 138670
2021/12/16 07:57:47 Waiting for src bucket item count to get to: 200000 curr count: 154967
2021/12/16 07:57:52 Waiting for src bucket item count to get to: 200000 curr count: 154967
2021/12/16 07:57:57 Waiting for src bucket item count to get to: 200000 curr count: 170194
2021/12/16 07:58:02 Waiting for src bucket item count to get to: 200000 curr count: 170194
2021/12/16 07:58:07 Waiting for src bucket item count to get to: 200000 curr count: 185791
2021/12/16 07:58:12 Waiting for src bucket item count to get to: 200000 curr count: 185791
2021/12/16 07:58:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 07:58:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 07:58:17 Starting bucket ops to source bucket
2021/12/16 08:02:07 Waiting for dst bucket item count to get to: 0 curr count: 22461
2021/12/16 08:02:12 Waiting for dst bucket item count to get to: 0 curr count: 4382
2021/12/16 08:02:17 Waiting for dst bucket item count to get to: 0 curr count: 4382
2021/12/16 08:02:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:02:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:02:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:02:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 08:02:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 08:02:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 08:02:37 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/16 08:02:37 App: TestEventingRebOutWithOps got un-deployed
2021/12/16 08:02:37 No eventing-con process running
2021/12/16 08:02:37 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/16 08:02:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:02:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:02:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:02:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:02:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:02:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (464.00s)
=== RUN   TestEventingSwapRebWithOps
2021/12/16 08:03:01 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 08:03:01 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 08:03:01 No eventing-con process running
2021/12/16 08:03:01 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/16 08:03:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:03:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:03:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:03:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:03:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:03:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:03:19 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:03:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 08:03:25 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 08:03:30 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 08:03:35 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 08:03:40 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/16 08:03:40 App: TestEventingSwapRebWithOps got deployed
2021/12/16 08:03:40 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 08:03:40 Starting bucket ops to source bucket
2021/12/16 08:03:41 Removing node(s): [] from the cluster
2021/12/16 08:03:46 Rebalance progress: 12.5
2021/12/16 08:03:51 Rebalance progress: 12.5
2021/12/16 08:03:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:03:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:03:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:03:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 08:03:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:04:02 Rebalance progress: 12.5
2021/12/16 08:04:07 Rebalance progress: 12.5
2021/12/16 08:04:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:04:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:04:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 08:04:17 Rebalance progress: 12.5
2021/12/16 08:04:22 Rebalance progress: 12.5
2021/12/16 08:04:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:04:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:04:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:04:27 Waiting for src bucket item count to get to: 200000 curr count: 59460
2021/12/16 08:04:32 Waiting for src bucket item count to get to: 200000 curr count: 74137
2021/12/16 08:04:37 Waiting for src bucket item count to get to: 200000 curr count: 74137
2021/12/16 08:04:42 Waiting for src bucket item count to get to: 200000 curr count: 89897
2021/12/16 08:04:47 Waiting for src bucket item count to get to: 200000 curr count: 89897
2021/12/16 08:04:52 Waiting for src bucket item count to get to: 200000 curr count: 105888
2021/12/16 08:04:57 Waiting for src bucket item count to get to: 200000 curr count: 105888
2021/12/16 08:05:02 Waiting for src bucket item count to get to: 200000 curr count: 121979
2021/12/16 08:05:07 Waiting for src bucket item count to get to: 200000 curr count: 121979
2021/12/16 08:05:12 Waiting for src bucket item count to get to: 200000 curr count: 137910
2021/12/16 08:05:17 Waiting for src bucket item count to get to: 200000 curr count: 137910
2021/12/16 08:05:22 Waiting for src bucket item count to get to: 200000 curr count: 153867
2021/12/16 08:05:27 Waiting for src bucket item count to get to: 200000 curr count: 153867
2021/12/16 08:05:32 Waiting for src bucket item count to get to: 200000 curr count: 168728
2021/12/16 08:05:37 Waiting for src bucket item count to get to: 200000 curr count: 168728
2021/12/16 08:05:42 Waiting for src bucket item count to get to: 200000 curr count: 184260
2021/12/16 08:05:47 Waiting for src bucket item count to get to: 200000 curr count: 184260
2021/12/16 08:05:52 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 08:05:52 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 08:05:52 Starting bucket ops to source bucket
2021/12/16 08:09:42 Waiting for dst bucket item count to get to: 0 curr count: 13921
2021/12/16 08:09:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:09:42 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:09:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:09:47 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 08:09:52 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 08:09:57 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 08:10:02 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/16 08:10:02 App: TestEventingSwapRebWithOps got un-deployed
2021/12/16 08:10:02 No eventing-con process running
2021/12/16 08:10:02 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/16 08:10:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:10:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:10:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:10:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:10:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:10:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (445.37s)
=== RUN   TestKVRebInWithOps
2021/12/16 08:10:26 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 08:10:26 App: TestKVRebInWithOps got un-deployed
2021/12/16 08:10:26 No eventing-con process running
2021/12/16 08:10:26 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/16 08:10:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:10:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:10:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:10:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:10:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:10:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:10:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:10:44 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:10:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 08:10:51 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 08:10:56 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 08:11:01 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 08:11:06 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/16 08:11:06 App: TestKVRebInWithOps got deployed
2021/12/16 08:11:06 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:11:06 Starting bucket ops to source bucket
2021/12/16 08:11:06 Removing node(s): [] from the cluster
2021/12/16 08:11:11 Rebalance progress: 12.5
2021/12/16 08:11:16 Rebalance progress: 12.5
2021/12/16 08:11:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:11:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:11:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:11:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:11:26 Rebalance progress: 12.5
2021/12/16 08:11:31 Rebalance progress: 12.5
2021/12/16 08:11:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:11:36 Waiting for src bucket item count to get to: 200000 curr count: 35661
2021/12/16 08:11:41 Waiting for src bucket item count to get to: 200000 curr count: 51428
2021/12/16 08:11:46 Waiting for src bucket item count to get to: 200000 curr count: 51428
2021/12/16 08:11:51 Waiting for src bucket item count to get to: 200000 curr count: 67573
2021/12/16 08:11:56 Waiting for src bucket item count to get to: 200000 curr count: 67573
2021/12/16 08:12:01 Waiting for src bucket item count to get to: 200000 curr count: 82389
2021/12/16 08:12:06 Waiting for src bucket item count to get to: 200000 curr count: 82389
2021/12/16 08:12:11 Waiting for src bucket item count to get to: 200000 curr count: 98523
2021/12/16 08:12:16 Waiting for src bucket item count to get to: 200000 curr count: 98523
2021/12/16 08:12:21 Waiting for src bucket item count to get to: 200000 curr count: 98523
2021/12/16 08:12:26 Waiting for src bucket item count to get to: 200000 curr count: 114585
2021/12/16 08:12:31 Waiting for src bucket item count to get to: 200000 curr count: 114585
2021/12/16 08:12:36 Waiting for src bucket item count to get to: 200000 curr count: 130477
2021/12/16 08:12:41 Waiting for src bucket item count to get to: 200000 curr count: 130477
2021/12/16 08:12:46 Waiting for src bucket item count to get to: 200000 curr count: 146364
2021/12/16 08:12:51 Waiting for src bucket item count to get to: 200000 curr count: 146364
2021/12/16 08:12:56 Waiting for src bucket item count to get to: 200000 curr count: 162407
2021/12/16 08:13:01 Waiting for src bucket item count to get to: 200000 curr count: 162407
2021/12/16 08:13:06 Waiting for src bucket item count to get to: 200000 curr count: 176953
2021/12/16 08:13:11 Waiting for src bucket item count to get to: 200000 curr count: 176953
2021/12/16 08:13:16 Waiting for src bucket item count to get to: 200000 curr count: 192826
2021/12/16 08:13:21 Waiting for src bucket item count to get to: 200000 curr count: 192826
2021/12/16 08:13:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 08:13:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 08:13:26 Starting bucket ops to source bucket
2021/12/16 08:17:16 Waiting for dst bucket item count to get to: 0 curr count: 3852
2021/12/16 08:17:21 Waiting for dst bucket item count to get to: 0 curr count: 3852
2021/12/16 08:17:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:17:21 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:17:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:17:27 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 08:17:32 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 08:17:37 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 08:17:42 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/16 08:17:42 App: TestKVRebInWithOps got un-deployed
2021/12/16 08:17:42 No eventing-con process running
2021/12/16 08:17:42 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/16 08:17:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:17:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:17:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:17:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:17:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.30s)
=== RUN   TestKVRebOutWithOps
2021/12/16 08:18:00 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:18:00 Removing node(s): [] from the cluster
2021/12/16 08:18:05 Rebalance progress: 100
2021/12/16 08:18:10 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 08:18:10 App: TestKVRebOutWithOps got un-deployed
2021/12/16 08:18:10 No eventing-con process running
2021/12/16 08:18:10 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/16 08:18:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:18:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:18:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:18:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:18:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:18:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:18:29 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:18:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 08:18:35 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 08:18:40 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 08:18:45 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 08:18:50 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/16 08:18:50 App: TestKVRebOutWithOps got deployed
2021/12/16 08:18:50 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:18:50 Starting bucket ops to source bucket
2021/12/16 08:18:55 Rebalance progress: 12.5
2021/12/16 08:19:00 Rebalance progress: 12.5
2021/12/16 08:19:05 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:19:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:19:05 Waiting for src bucket item count to get to: 200000 curr count: 13485
2021/12/16 08:19:10 Waiting for src bucket item count to get to: 200000 curr count: 28971
2021/12/16 08:19:15 Waiting for src bucket item count to get to: 200000 curr count: 28971
2021/12/16 08:19:20 Waiting for src bucket item count to get to: 200000 curr count: 45203
2021/12/16 08:19:25 Waiting for src bucket item count to get to: 200000 curr count: 45203
2021/12/16 08:19:30 Waiting for src bucket item count to get to: 200000 curr count: 45203
2021/12/16 08:19:35 Waiting for src bucket item count to get to: 200000 curr count: 61374
2021/12/16 08:19:40 Waiting for src bucket item count to get to: 200000 curr count: 61374
2021/12/16 08:19:45 Waiting for src bucket item count to get to: 200000 curr count: 76495
2021/12/16 08:19:50 Waiting for src bucket item count to get to: 200000 curr count: 76495
2021/12/16 08:19:55 Waiting for src bucket item count to get to: 200000 curr count: 92058
2021/12/16 08:20:00 Waiting for src bucket item count to get to: 200000 curr count: 92058
2021/12/16 08:20:05 Waiting for src bucket item count to get to: 200000 curr count: 108153
2021/12/16 08:20:10 Waiting for src bucket item count to get to: 200000 curr count: 108153
2021/12/16 08:20:15 Waiting for src bucket item count to get to: 200000 curr count: 124071
2021/12/16 08:20:20 Waiting for src bucket item count to get to: 200000 curr count: 124071
2021/12/16 08:20:25 Waiting for src bucket item count to get to: 200000 curr count: 140025
2021/12/16 08:20:30 Waiting for src bucket item count to get to: 200000 curr count: 140025
2021/12/16 08:20:35 Waiting for src bucket item count to get to: 200000 curr count: 156287
2021/12/16 08:20:40 Waiting for src bucket item count to get to: 200000 curr count: 156287
2021/12/16 08:20:45 Waiting for src bucket item count to get to: 200000 curr count: 171466
2021/12/16 08:20:50 Waiting for src bucket item count to get to: 200000 curr count: 171466
2021/12/16 08:20:55 Waiting for src bucket item count to get to: 200000 curr count: 187373
2021/12/16 08:21:00 Waiting for src bucket item count to get to: 200000 curr count: 187373
2021/12/16 08:21:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 08:21:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 08:21:05 Starting bucket ops to source bucket
2021/12/16 08:24:56 Waiting for dst bucket item count to get to: 0 curr count: 2257
2021/12/16 08:25:01 Waiting for dst bucket item count to get to: 0 curr count: 2257
2021/12/16 08:25:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:25:01 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:25:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:25:06 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 08:25:11 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 08:25:16 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 08:25:21 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/16 08:25:21 App: TestKVRebOutWithOps got un-deployed
2021/12/16 08:25:21 No eventing-con process running
2021/12/16 08:25:21 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/16 08:25:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:25:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:25:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:25:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (459.10s)
=== RUN   TestKVSwapRebWithOps
2021/12/16 08:25:44 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 08:25:44 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 08:25:44 No eventing-con process running
2021/12/16 08:25:44 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/16 08:25:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:25:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:25:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:25:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:25:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:25:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:26:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:26:03 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:26:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/16 08:26:09 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 08:26:14 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 08:26:19 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 08:26:24 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/16 08:26:24 App: TestKVSwapRebWithOps got deployed
2021/12/16 08:26:24 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:26:24 Starting bucket ops to source bucket
2021/12/16 08:26:24 Removing node(s): [] from the cluster
2021/12/16 08:26:30 Rebalance progress: 12.5
2021/12/16 08:26:35 Rebalance progress: 12.5
2021/12/16 08:26:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:26:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:26:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:26:40 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 08:26:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:26:45 Rebalance progress: 12.5
2021/12/16 08:26:50 Rebalance progress: 12.5
2021/12/16 08:26:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:26:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:26:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:26:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 08:27:00 Rebalance progress: 12.5
2021/12/16 08:27:05 Rebalance progress: 12.5
2021/12/16 08:27:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:27:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:27:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:27:10 Waiting for src bucket item count to get to: 200000 curr count: 53417
2021/12/16 08:27:15 Waiting for src bucket item count to get to: 200000 curr count: 69502
2021/12/16 08:27:20 Waiting for src bucket item count to get to: 200000 curr count: 69502
2021/12/16 08:27:25 Waiting for src bucket item count to get to: 200000 curr count: 83965
2021/12/16 08:27:30 Waiting for src bucket item count to get to: 200000 curr count: 83965
2021/12/16 08:27:35 Waiting for src bucket item count to get to: 200000 curr count: 100073
2021/12/16 08:27:40 Waiting for src bucket item count to get to: 200000 curr count: 116324
2021/12/16 08:27:45 Waiting for src bucket item count to get to: 200000 curr count: 116324
2021/12/16 08:27:50 Waiting for src bucket item count to get to: 200000 curr count: 132591
2021/12/16 08:27:55 Waiting for src bucket item count to get to: 200000 curr count: 132591
2021/12/16 08:28:00 Waiting for src bucket item count to get to: 200000 curr count: 148886
2021/12/16 08:28:05 Waiting for src bucket item count to get to: 200000 curr count: 148886
2021/12/16 08:28:10 Waiting for src bucket item count to get to: 200000 curr count: 148886
2021/12/16 08:28:15 Waiting for src bucket item count to get to: 200000 curr count: 164992
2021/12/16 08:28:20 Waiting for src bucket item count to get to: 200000 curr count: 164992
2021/12/16 08:28:25 Waiting for src bucket item count to get to: 200000 curr count: 179597
2021/12/16 08:28:30 Waiting for src bucket item count to get to: 200000 curr count: 179597
2021/12/16 08:28:35 Waiting for src bucket item count to get to: 200000 curr count: 195817
2021/12/16 08:28:40 Waiting for src bucket item count to get to: 200000 curr count: 195817
2021/12/16 08:28:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/16 08:28:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/16 08:28:45 Starting bucket ops to source bucket
2021/12/16 08:32:35 Waiting for dst bucket item count to get to: 0 curr count: 19615
2021/12/16 08:32:40 Waiting for dst bucket item count to get to: 0 curr count: 1288
2021/12/16 08:32:45 Waiting for dst bucket item count to get to: 0 curr count: 1288
2021/12/16 08:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:32:45 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:32:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:32:50 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 08:32:55 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 08:33:00 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 08:33:05 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/16 08:33:05 App: TestKVSwapRebWithOps got un-deployed
2021/12/16 08:33:05 No eventing-con process running
2021/12/16 08:33:05 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/16 08:33:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:33:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:33:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:33:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:33:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:33:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (464.51s)
=== RUN   TestEventingRebInWithTimers
2021/12/16 08:33:29 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 08:33:29 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 08:33:29 No eventing-con process running
2021/12/16 08:33:29 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/16 08:33:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:33:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:33:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:33:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:33:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:33:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:33:47 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:33:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:33:54 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 08:33:59 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 08:34:04 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 08:34:09 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/16 08:34:09 App: TestEventingRebInWithTimers got deployed
2021/12/16 08:34:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 08:34:09 Starting bucket ops to source bucket
2021/12/16 08:34:10 Removing node(s): [] from the cluster
2021/12/16 08:34:15 Rebalance progress: 12.5
2021/12/16 08:34:20 Rebalance progress: 12.5
2021/12/16 08:34:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:34:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:34:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:34:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:34:30 Rebalance progress: 12.5
2021/12/16 08:34:35 Rebalance progress: 12.5
2021/12/16 08:34:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:34:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:34:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:34:40 Waiting for src bucket item count to get to: 50000 curr count: 25969
2021/12/16 08:34:45 Waiting for src bucket item count to get to: 50000 curr count: 35746
2021/12/16 08:34:50 Waiting for src bucket item count to get to: 50000 curr count: 35746
2021/12/16 08:34:55 Waiting for src bucket item count to get to: 50000 curr count: 47771
2021/12/16 08:35:00 Waiting for src bucket item count to get to: 50000 curr count: 47771
2021/12/16 08:35:10 Waiting for dst bucket item count to get to: 50000 curr count: 46644
2021/12/16 08:35:15 Waiting for dst bucket item count to get to: 50000 curr count: 46644
2021/12/16 08:35:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:35:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:35:15 Starting bucket ops to source bucket
2021/12/16 08:36:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:36:30 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:36:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:36:40 Waiting for dst bucket item count to get to: 0 curr count: 44106
2021/12/16 08:36:45 Waiting for dst bucket item count to get to: 0 curr count: 44106
2021/12/16 08:36:50 Waiting for dst bucket item count to get to: 0 curr count: 28622
2021/12/16 08:36:55 Waiting for dst bucket item count to get to: 0 curr count: 28622
2021/12/16 08:37:00 Waiting for dst bucket item count to get to: 0 curr count: 14856
2021/12/16 08:37:05 Waiting for dst bucket item count to get to: 0 curr count: 14856
2021/12/16 08:37:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:37:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:37:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:37:10 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 08:37:15 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 08:37:21 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 08:37:26 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/16 08:37:26 App: TestEventingRebInWithTimers got un-deployed
2021/12/16 08:37:26 No eventing-con process running
2021/12/16 08:37:26 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/16 08:37:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:37:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:37:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:37:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:37:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (260.36s)
=== RUN   TestEventingRebOutWithTimers
2021/12/16 08:37:44 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 08:37:44 Removing node(s): [] from the cluster
2021/12/16 08:37:50 Rebalance progress: 100
2021/12/16 08:37:55 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 08:37:55 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 08:37:55 No eventing-con process running
2021/12/16 08:37:55 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/16 08:37:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:38:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:38:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:38:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:38:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:38:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:38:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:38:13 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:38:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:38:19 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 08:38:24 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 08:38:29 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 08:38:34 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/16 08:38:34 App: TestEventingRebOutWithTimers got deployed
2021/12/16 08:38:34 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:38:34 Starting bucket ops to source bucket
2021/12/16 08:38:39 Rebalance progress: 12.5
2021/12/16 08:38:44 Rebalance progress: 12.5
2021/12/16 08:38:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:38:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:38:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:38:49 Waiting for src bucket item count to get to: 50000 curr count: 6219
2021/12/16 08:38:54 Waiting for src bucket item count to get to: 50000 curr count: 19129
2021/12/16 08:38:59 Waiting for src bucket item count to get to: 50000 curr count: 19129
2021/12/16 08:39:04 Waiting for src bucket item count to get to: 50000 curr count: 30039
2021/12/16 08:39:09 Waiting for src bucket item count to get to: 50000 curr count: 30039
2021/12/16 08:39:14 Waiting for src bucket item count to get to: 50000 curr count: 41613
2021/12/16 08:39:19 Waiting for src bucket item count to get to: 50000 curr count: 41613
2021/12/16 08:39:24 Waiting for src bucket item count to get to: 50000 curr count: 41613
2021/12/16 08:39:34 Waiting for dst bucket item count to get to: 50000 curr count: 39532
2021/12/16 08:39:39 Waiting for dst bucket item count to get to: 50000 curr count: 47564
2021/12/16 08:39:44 Waiting for dst bucket item count to get to: 50000 curr count: 47564
2021/12/16 08:39:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:39:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:39:44 Starting bucket ops to source bucket
2021/12/16 08:40:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:41:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:41:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:41:10 Waiting for dst bucket item count to get to: 0 curr count: 48760
2021/12/16 08:41:15 Waiting for dst bucket item count to get to: 0 curr count: 48760
2021/12/16 08:41:20 Waiting for dst bucket item count to get to: 0 curr count: 31114
2021/12/16 08:41:25 Waiting for dst bucket item count to get to: 0 curr count: 31114
2021/12/16 08:41:30 Waiting for dst bucket item count to get to: 0 curr count: 11113
2021/12/16 08:41:35 Waiting for dst bucket item count to get to: 0 curr count: 11113
2021/12/16 08:41:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:41:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:41:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:41:40 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 08:41:45 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 08:41:50 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 08:41:55 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/16 08:41:55 App: TestEventingRebOutWithTimers got un-deployed
2021/12/16 08:41:55 No eventing-con process running
2021/12/16 08:41:55 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/16 08:41:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:42:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:42:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:42:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:42:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.67s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/16 08:42:19 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 08:42:19 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 08:42:19 No eventing-con process running
2021/12/16 08:42:19 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/16 08:42:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:42:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:42:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:42:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:42:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:42:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:42:37 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:42:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:42:44 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 08:42:49 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 08:42:54 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 08:42:59 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/16 08:42:59 App: TestEventingSwapRebWithTimers got deployed
2021/12/16 08:42:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 08:42:59 Starting bucket ops to source bucket
2021/12/16 08:42:59 Removing node(s): [] from the cluster
2021/12/16 08:43:04 Rebalance progress: 12.5
2021/12/16 08:43:09 Rebalance progress: 12.5
2021/12/16 08:43:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:43:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 08:43:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:43:19 Rebalance progress: 12.5
2021/12/16 08:43:24 Rebalance progress: 12.5
2021/12/16 08:43:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:43:29 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 08:43:34 Rebalance progress: 12.5
2021/12/16 08:43:39 Rebalance progress: 12.5
2021/12/16 08:43:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:43:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:43:44 Waiting for src bucket item count to get to: 50000 curr count: 47313
2021/12/16 08:43:50 Waiting for src bucket item count to get to: 50000 curr count: 47313
2021/12/16 08:44:00 Waiting for dst bucket item count to get to: 50000 curr count: 48161
2021/12/16 08:44:05 Waiting for dst bucket item count to get to: 50000 curr count: 48161
2021/12/16 08:44:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:44:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:44:05 Starting bucket ops to source bucket
2021/12/16 08:45:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:45:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:45:25 Waiting for dst bucket item count to get to: 0 curr count: 39701
2021/12/16 08:45:30 Waiting for dst bucket item count to get to: 0 curr count: 39701
2021/12/16 08:45:35 Waiting for dst bucket item count to get to: 0 curr count: 39701
2021/12/16 08:45:40 Waiting for dst bucket item count to get to: 0 curr count: 24979
2021/12/16 08:45:45 Waiting for dst bucket item count to get to: 0 curr count: 24979
2021/12/16 08:45:50 Waiting for dst bucket item count to get to: 0 curr count: 11415
2021/12/16 08:45:55 Waiting for dst bucket item count to get to: 0 curr count: 11415
2021/12/16 08:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:45:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:45:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:46:00 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 08:46:06 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 08:46:11 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 08:46:16 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/16 08:46:16 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/16 08:46:16 No eventing-con process running
2021/12/16 08:46:16 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/16 08:46:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:46:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:46:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (260.36s)
=== RUN   TestKVRebInWithTimers
2021/12/16 08:46:39 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 08:46:39 App: TestKVRebInWithTimers got un-deployed
2021/12/16 08:46:39 No eventing-con process running
2021/12/16 08:46:39 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/16 08:46:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:46:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:46:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:46:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:46:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:46:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:46:58 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:46:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:47:04 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 08:47:09 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 08:47:14 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 08:47:19 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/16 08:47:19 App: TestKVRebInWithTimers got deployed
2021/12/16 08:47:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:47:19 Starting bucket ops to source bucket
2021/12/16 08:47:20 Removing node(s): [] from the cluster
2021/12/16 08:47:25 Rebalance progress: 12.5
2021/12/16 08:47:30 Rebalance progress: 12.5
2021/12/16 08:47:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:47:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:47:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:47:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:47:40 Rebalance progress: 12.5
2021/12/16 08:47:45 Rebalance progress: 12.5
2021/12/16 08:47:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:47:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:47:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:47:50 Waiting for src bucket item count to get to: 50000 curr count: 25365
2021/12/16 08:47:55 Waiting for src bucket item count to get to: 50000 curr count: 35038
2021/12/16 08:48:00 Waiting for src bucket item count to get to: 50000 curr count: 35038
2021/12/16 08:48:05 Waiting for src bucket item count to get to: 50000 curr count: 47209
2021/12/16 08:48:10 Waiting for src bucket item count to get to: 50000 curr count: 47209
2021/12/16 08:48:20 Waiting for dst bucket item count to get to: 50000 curr count: 47469
2021/12/16 08:48:25 Waiting for dst bucket item count to get to: 50000 curr count: 47469
2021/12/16 08:48:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:48:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:48:25 Starting bucket ops to source bucket
2021/12/16 08:49:35 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:49:40 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:49:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:49:50 Waiting for dst bucket item count to get to: 0 curr count: 44455
2021/12/16 08:49:55 Waiting for dst bucket item count to get to: 0 curr count: 44455
2021/12/16 08:50:00 Waiting for dst bucket item count to get to: 0 curr count: 26655
2021/12/16 08:50:05 Waiting for dst bucket item count to get to: 0 curr count: 26655
2021/12/16 08:50:10 Waiting for dst bucket item count to get to: 0 curr count: 14362
2021/12/16 08:50:15 Waiting for dst bucket item count to get to: 0 curr count: 14362
2021/12/16 08:50:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:50:15 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:50:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:50:21 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 08:50:26 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 08:50:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 08:50:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/16 08:50:36 App: TestKVRebInWithTimers got un-deployed
2021/12/16 08:50:36 No eventing-con process running
2021/12/16 08:50:36 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/16 08:50:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:50:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:50:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:50:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:50:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:50:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (260.10s)
=== RUN   TestKVRebOutWithTimers
2021/12/16 08:50:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:50:55 Removing node(s): [] from the cluster
2021/12/16 08:51:00 Rebalance progress: 100
2021/12/16 08:51:05 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 08:51:05 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 08:51:05 No eventing-con process running
2021/12/16 08:51:05 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/16 08:51:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:51:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:51:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:51:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:51:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:51:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:51:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:51:23 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:51:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:51:29 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 08:51:34 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 08:51:39 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 08:51:44 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/16 08:51:44 App: TestKVRebOutWithTimers got deployed
2021/12/16 08:51:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:51:44 Starting bucket ops to source bucket
2021/12/16 08:51:50 Rebalance progress: 12.5
2021/12/16 08:51:55 Rebalance progress: 12.5
2021/12/16 08:52:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:52:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:52:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:52:00 Waiting for src bucket item count to get to: 50000 curr count: 6108
2021/12/16 08:52:05 Waiting for src bucket item count to get to: 50000 curr count: 19724
2021/12/16 08:52:10 Waiting for src bucket item count to get to: 50000 curr count: 19724
2021/12/16 08:52:15 Waiting for src bucket item count to get to: 50000 curr count: 30033
2021/12/16 08:52:20 Waiting for src bucket item count to get to: 50000 curr count: 42073
2021/12/16 08:52:25 Waiting for src bucket item count to get to: 50000 curr count: 42073
2021/12/16 08:52:30 Waiting for src bucket item count to get to: 50000 curr count: 42073
2021/12/16 08:52:40 Waiting for dst bucket item count to get to: 50000 curr count: 41276
2021/12/16 08:52:45 Waiting for dst bucket item count to get to: 50000 curr count: 41276
2021/12/16 08:52:50 Waiting for dst bucket item count to get to: 50000 curr count: 49503
2021/12/16 08:52:55 Waiting for dst bucket item count to get to: 50000 curr count: 49503
2021/12/16 08:52:55 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:52:55 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:52:55 Starting bucket ops to source bucket
2021/12/16 08:54:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:54:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:54:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:54:21 Waiting for dst bucket item count to get to: 0 curr count: 46055
2021/12/16 08:54:26 Waiting for dst bucket item count to get to: 0 curr count: 46055
2021/12/16 08:54:31 Waiting for dst bucket item count to get to: 0 curr count: 29000
2021/12/16 08:54:36 Waiting for dst bucket item count to get to: 0 curr count: 29000
2021/12/16 08:54:41 Waiting for dst bucket item count to get to: 0 curr count: 10315
2021/12/16 08:54:46 Waiting for dst bucket item count to get to: 0 curr count: 10315
2021/12/16 08:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:54:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:54:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:54:51 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 08:54:56 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 08:55:01 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 08:55:06 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/16 08:55:06 App: TestKVRebOutWithTimers got un-deployed
2021/12/16 08:55:06 No eventing-con process running
2021/12/16 08:55:06 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/16 08:55:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:55:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:55:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:55:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:55:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (270.23s)
=== RUN   TestKVSwapRebWithTimers
2021/12/16 08:55:30 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:55:30 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 08:55:30 No eventing-con process running
2021/12/16 08:55:30 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/16 08:55:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:55:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:55:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:55:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:55:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:55:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:55:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:55:48 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/16 08:55:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/16 08:55:54 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 08:55:59 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 08:56:04 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 08:56:09 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/16 08:56:09 App: TestKVSwapRebWithTimers got deployed
2021/12/16 08:56:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 08:56:09 Starting bucket ops to source bucket
2021/12/16 08:56:10 Removing node(s): [] from the cluster
2021/12/16 08:56:15 Rebalance progress: 12.5
2021/12/16 08:56:20 Rebalance progress: 12.5
2021/12/16 08:56:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:56:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 08:56:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 08:56:30 Rebalance progress: 12.5
2021/12/16 08:56:35 Rebalance progress: 12.5
2021/12/16 08:56:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:56:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 08:56:45 Rebalance progress: 12.5
2021/12/16 08:56:50 Rebalance progress: 12.5
2021/12/16 08:56:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:56:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 08:56:56 Waiting for src bucket item count to get to: 50000 curr count: 45556
2021/12/16 08:57:01 Waiting for src bucket item count to get to: 50000 curr count: 45556
2021/12/16 08:57:11 Waiting for dst bucket item count to get to: 50000 curr count: 44662
2021/12/16 08:57:16 Waiting for dst bucket item count to get to: 50000 curr count: 49574
2021/12/16 08:57:21 Waiting for dst bucket item count to get to: 50000 curr count: 49574
2021/12/16 08:57:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 08:57:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 08:57:21 Starting bucket ops to source bucket
2021/12/16 08:58:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:58:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 08:58:41 Waiting for dst bucket item count to get to: 0 curr count: 47518
2021/12/16 08:58:46 Waiting for dst bucket item count to get to: 0 curr count: 47518
2021/12/16 08:58:51 Waiting for dst bucket item count to get to: 0 curr count: 34428
2021/12/16 08:58:56 Waiting for dst bucket item count to get to: 0 curr count: 34428
2021/12/16 08:59:01 Waiting for dst bucket item count to get to: 0 curr count: 23035
2021/12/16 08:59:06 Waiting for dst bucket item count to get to: 0 curr count: 23035
2021/12/16 08:59:11 Waiting for dst bucket item count to get to: 0 curr count: 1554
2021/12/16 08:59:16 Waiting for dst bucket item count to get to: 0 curr count: 1554
2021/12/16 08:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:59:16 Post DELETE ops. Expected item count: 0 got 0
2021/12/16 08:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 08:59:21 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:26 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:31 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:36 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:41 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:46 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/16 08:59:46 App: TestKVSwapRebWithTimers got un-deployed
2021/12/16 08:59:46 No eventing-con process running
2021/12/16 08:59:46 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/16 08:59:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 08:59:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 08:59:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 08:59:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:00:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:00:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (280.16s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/16 09:00:10 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:00:10 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 09:00:10 No eventing-con process running
2021/12/16 09:00:10 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/16 09:00:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:00:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:00:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:00:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:00:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:00:29 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:00:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 09:00:35 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 09:00:40 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 09:00:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 09:00:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/16 09:00:50 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/16 09:00:50 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:00:50 Starting bucket ops to source bucket
2021/12/16 09:00:51 Removing node(s): [] from the cluster
2021/12/16 09:00:56 Rebalance progress: 12.5
2021/12/16 09:01:01 Rebalance progress: 12.5
2021/12/16 09:01:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:01:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:01:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:01:11 Rebalance progress: 12.5
2021/12/16 09:01:16 Rebalance progress: 12.5
2021/12/16 09:01:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:01:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:01:21 Waiting for src bucket item count to get to: 50000 curr count: 34957
2021/12/16 09:01:26 Waiting for src bucket item count to get to: 50000 curr count: 34957
2021/12/16 09:01:31 Waiting for src bucket item count to get to: 50000 curr count: 44141
2021/12/16 09:01:36 Waiting for src bucket item count to get to: 50000 curr count: 44141
2021/12/16 09:01:46 Waiting for dst bucket item count to get to: 50000 curr count: 34941
2021/12/16 09:01:51 Waiting for dst bucket item count to get to: 50000 curr count: 34941
2021/12/16 09:01:56 Waiting for dst bucket item count to get to: 50000 curr count: 45781
2021/12/16 09:02:01 Waiting for dst bucket item count to get to: 50000 curr count: 45781
2021/12/16 09:02:01 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:02:01 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:02:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:02:06 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:11 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:16 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:21 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:27 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:32 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/16 09:02:32 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/16 09:02:32 No eventing-con process running
2021/12/16 09:02:32 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/16 09:02:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:02:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:02:43 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:02:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:02:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:02:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:02:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:03:00 Waiting for dst bucket item count to get to: 0 curr count: 11678
2021/12/16 09:03:05 Waiting for dst bucket item count to get to: 0 curr count: 11678
2021/12/16 09:03:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (185.45s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/16 09:03:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:03:10 Removing node(s): [] from the cluster
2021/12/16 09:03:15 Rebalance progress: 100
2021/12/16 09:03:20 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:03:20 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 09:03:20 No eventing-con process running
2021/12/16 09:03:20 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/16 09:03:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:03:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:03:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:03:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:03:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:03:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:03:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:03:39 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:03:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 09:03:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 09:03:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 09:03:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 09:04:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/16 09:04:00 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/16 09:04:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:04:00 Starting bucket ops to source bucket
2021/12/16 09:04:05 Rebalance progress: 12.5
2021/12/16 09:04:10 Rebalance progress: 12.5
2021/12/16 09:04:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:04:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:04:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:04:15 Waiting for src bucket item count to get to: 50000 curr count: 12025
2021/12/16 09:04:20 Waiting for src bucket item count to get to: 50000 curr count: 26205
2021/12/16 09:04:25 Waiting for src bucket item count to get to: 50000 curr count: 26205
2021/12/16 09:04:30 Waiting for src bucket item count to get to: 50000 curr count: 37133
2021/12/16 09:04:35 Waiting for src bucket item count to get to: 50000 curr count: 37133
2021/12/16 09:04:40 Waiting for src bucket item count to get to: 50000 curr count: 46655
2021/12/16 09:04:45 Waiting for src bucket item count to get to: 50000 curr count: 46655
2021/12/16 09:04:50 Waiting for src bucket item count to get to: 50000 curr count: 46655
2021/12/16 09:05:00 Waiting for dst bucket item count to get to: 50000 curr count: 36573
2021/12/16 09:05:05 Waiting for dst bucket item count to get to: 50000 curr count: 36573
2021/12/16 09:05:10 Waiting for dst bucket item count to get to: 50000 curr count: 46919
2021/12/16 09:05:15 Waiting for dst bucket item count to get to: 50000 curr count: 46919
2021/12/16 09:05:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:05:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:05:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:05:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:26 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:31 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:36 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:46 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/16 09:05:46 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/16 09:05:46 No eventing-con process running
2021/12/16 09:05:46 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/16 09:05:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:05:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:05:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:05:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:05:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:06:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:06:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.16s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/16 09:06:14 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:06:14 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 09:06:14 No eventing-con process running
2021/12/16 09:06:14 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/16 09:06:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:06:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:06:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:06:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:06:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:06:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:06:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:06:33 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:06:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 09:06:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 09:06:44 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 09:06:49 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 09:06:54 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/16 09:06:54 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/16 09:06:54 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:06:54 Starting bucket ops to source bucket
2021/12/16 09:06:54 Removing node(s): [] from the cluster
2021/12/16 09:07:00 Rebalance progress: 12.5
2021/12/16 09:07:05 Rebalance progress: 12.5
2021/12/16 09:07:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:07:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 09:07:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:07:15 Rebalance progress: 12.5
2021/12/16 09:07:20 Rebalance progress: 12.5
2021/12/16 09:07:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:07:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:07:31 Rebalance progress: 12.5
2021/12/16 09:07:36 Rebalance progress: 12.5
2021/12/16 09:07:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:07:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:07:41 Waiting for src bucket item count to get to: 50000 curr count: 40565
2021/12/16 09:07:51 Waiting for dst bucket item count to get to: 50000 curr count: 32744
2021/12/16 09:07:56 Waiting for dst bucket item count to get to: 50000 curr count: 32744
2021/12/16 09:08:01 Waiting for dst bucket item count to get to: 50000 curr count: 38777
2021/12/16 09:08:06 Waiting for dst bucket item count to get to: 50000 curr count: 38777
2021/12/16 09:08:06 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:08:06 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:08:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:08:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:21 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:31 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/16 09:08:36 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/16 09:08:36 No eventing-con process running
2021/12/16 09:08:36 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/16 09:08:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:08:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:08:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:08:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:08:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:08:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:08:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (170.14s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/16 09:09:04 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:09:04 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 09:09:04 No eventing-con process running
2021/12/16 09:09:04 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/16 09:09:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:09:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:09:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:09:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:09:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:09:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:09:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:09:23 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:09:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 09:09:29 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 09:09:34 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 09:09:39 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 09:09:44 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/16 09:09:44 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/16 09:09:44 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:09:44 Starting bucket ops to source bucket
2021/12/16 09:09:45 Removing node(s): [] from the cluster
2021/12/16 09:09:50 Rebalance progress: 12.5
2021/12/16 09:09:55 Rebalance progress: 12.5
2021/12/16 09:10:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:10:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:10:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:10:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:10:05 Rebalance progress: 12.5
2021/12/16 09:10:10 Rebalance progress: 12.5
2021/12/16 09:10:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:10:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:10:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:10:15 Waiting for src bucket item count to get to: 50000 curr count: 32089
2021/12/16 09:10:20 Waiting for src bucket item count to get to: 50000 curr count: 32089
2021/12/16 09:10:25 Waiting for src bucket item count to get to: 50000 curr count: 42664
2021/12/16 09:10:30 Waiting for src bucket item count to get to: 50000 curr count: 42664
2021/12/16 09:10:40 Waiting for dst bucket item count to get to: 50000 curr count: 30986
2021/12/16 09:10:45 Waiting for dst bucket item count to get to: 50000 curr count: 30986
2021/12/16 09:10:50 Waiting for dst bucket item count to get to: 50000 curr count: 37551
2021/12/16 09:10:50 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:10:50 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:10:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:10:55 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/16 09:11:20 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/16 09:11:20 No eventing-con process running
2021/12/16 09:11:20 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/16 09:11:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:11:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:11:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:11:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:11:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:11:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:11:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:11:49 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:11:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (174.32s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/16 09:11:54 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:11:54 Removing node(s): [] from the cluster
2021/12/16 09:11:59 Rebalance progress: 100
2021/12/16 09:12:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:12:04 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 09:12:04 No eventing-con process running
2021/12/16 09:12:04 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/16 09:12:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:12:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:12:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:12:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:12:23 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:12:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 09:12:29 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 09:12:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 09:12:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 09:12:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/16 09:12:44 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/16 09:12:44 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:12:44 Starting bucket ops to source bucket
2021/12/16 09:12:49 Rebalance progress: 12.5
2021/12/16 09:12:54 Rebalance progress: 12.5
2021/12/16 09:12:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:12:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:12:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:12:59 Waiting for src bucket item count to get to: 50000 curr count: 6277
2021/12/16 09:13:04 Waiting for src bucket item count to get to: 50000 curr count: 20035
2021/12/16 09:13:09 Waiting for src bucket item count to get to: 50000 curr count: 20035
2021/12/16 09:13:14 Waiting for src bucket item count to get to: 50000 curr count: 32137
2021/12/16 09:13:19 Waiting for src bucket item count to get to: 50000 curr count: 32137
2021/12/16 09:13:24 Waiting for src bucket item count to get to: 50000 curr count: 32137
2021/12/16 09:13:29 Waiting for src bucket item count to get to: 50000 curr count: 42907
2021/12/16 09:13:34 Waiting for src bucket item count to get to: 50000 curr count: 42907
2021/12/16 09:13:44 Waiting for dst bucket item count to get to: 50000 curr count: 33861
2021/12/16 09:13:49 Waiting for dst bucket item count to get to: 50000 curr count: 33861
2021/12/16 09:13:54 Waiting for dst bucket item count to get to: 50000 curr count: 41280
2021/12/16 09:13:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:13:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:13:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:13:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:09 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:14 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:19 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:25 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/16 09:14:25 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/16 09:14:25 No eventing-con process running
2021/12/16 09:14:25 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/16 09:14:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:14:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:14:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:14:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:14:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:14:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:14:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (179.40s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/16 09:14:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:14:58 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 09:14:58 No eventing-con process running
2021/12/16 09:14:58 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/16 09:15:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:15:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:15:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:15:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:15:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:15:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:15:17 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:15:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 09:15:23 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 09:15:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 09:15:33 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 09:15:38 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/16 09:15:38 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/16 09:15:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:15:38 Starting bucket ops to source bucket
2021/12/16 09:15:39 Removing node(s): [] from the cluster
2021/12/16 09:15:44 Rebalance progress: 12.5
2021/12/16 09:15:49 Rebalance progress: 12.5
2021/12/16 09:15:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:15:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:15:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:15:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:15:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:15:59 Rebalance progress: 12.5
2021/12/16 09:16:04 Rebalance progress: 12.5
2021/12/16 09:16:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:16:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:16:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:16:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:16:14 Rebalance progress: 12.5
2021/12/16 09:16:19 Rebalance progress: 12.5
2021/12/16 09:16:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:16:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:16:24 Waiting for src bucket item count to get to: 50000 curr count: 49613
2021/12/16 09:16:29 Waiting for src bucket item count to get to: 50000 curr count: 49613
2021/12/16 09:16:39 Waiting for dst bucket item count to get to: 50000 curr count: 35863
2021/12/16 09:16:44 Waiting for dst bucket item count to get to: 50000 curr count: 35863
2021/12/16 09:16:49 Waiting for dst bucket item count to get to: 50000 curr count: 42471
2021/12/16 09:16:54 Waiting for dst bucket item count to get to: 50000 curr count: 42471
2021/12/16 09:16:54 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:16:54 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:16:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:16:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:14 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:19 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:24 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/16 09:17:24 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/16 09:17:24 No eventing-con process running
2021/12/16 09:17:25 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/16 09:17:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:17:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:17:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:17:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:17:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:17:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:17:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:17:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.98s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/16 09:17:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:17:58 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 09:17:58 No eventing-con process running
2021/12/16 09:17:58 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/16 09:18:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:18:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:18:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:18:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:18:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:18:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:18:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:18:17 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:18:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 09:18:23 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 09:18:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 09:18:33 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 09:18:38 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/16 09:18:38 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/16 09:18:38 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:18:38 Starting bucket ops to source bucket
2021/12/16 09:18:39 Removing node(s): [] from the cluster
2021/12/16 09:18:44 Rebalance progress: 12.5
2021/12/16 09:18:49 Rebalance progress: 12.5
2021/12/16 09:18:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:18:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:18:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:18:59 Rebalance progress: 12.5
2021/12/16 09:19:04 Rebalance progress: 12.5
2021/12/16 09:19:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:19:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:19:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:19:09 Waiting for src bucket item count to get to: 50000 curr count: 27097
2021/12/16 09:19:14 Waiting for src bucket item count to get to: 50000 curr count: 36254
2021/12/16 09:19:19 Waiting for src bucket item count to get to: 50000 curr count: 36254
2021/12/16 09:19:24 Waiting for src bucket item count to get to: 50000 curr count: 49102
2021/12/16 09:19:29 Waiting for src bucket item count to get to: 50000 curr count: 49102
2021/12/16 09:19:39 Waiting for dst bucket item count to get to: 50000 curr count: 43580
2021/12/16 09:19:44 Waiting for dst bucket item count to get to: 50000 curr count: 43580
2021/12/16 09:19:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:19:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:19:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:19:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:19:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:19:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:20:04 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:20:09 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:20:14 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/16 09:20:14 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/16 09:20:14 No eventing-con process running
2021/12/16 09:20:14 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/16 09:20:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:20:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:20:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:20:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:20:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.54s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/16 09:20:43 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:20:44 Removing node(s): [] from the cluster
2021/12/16 09:20:49 Rebalance progress: 100
2021/12/16 09:20:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:20:54 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 09:20:54 No eventing-con process running
2021/12/16 09:20:54 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/16 09:20:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:21:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:21:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:21:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:21:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:21:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:21:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:21:12 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:21:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 09:21:18 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 09:21:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 09:21:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 09:21:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/16 09:21:33 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/16 09:21:33 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:21:33 Starting bucket ops to source bucket
2021/12/16 09:21:38 Rebalance progress: 12.5
2021/12/16 09:21:43 Rebalance progress: 12.5
2021/12/16 09:21:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:21:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:21:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:21:48 Waiting for src bucket item count to get to: 50000 curr count: 7511
2021/12/16 09:21:53 Waiting for src bucket item count to get to: 50000 curr count: 21408
2021/12/16 09:21:58 Waiting for src bucket item count to get to: 50000 curr count: 21408
2021/12/16 09:22:03 Waiting for src bucket item count to get to: 50000 curr count: 32847
2021/12/16 09:22:08 Waiting for src bucket item count to get to: 50000 curr count: 32847
2021/12/16 09:22:13 Waiting for src bucket item count to get to: 50000 curr count: 43368
2021/12/16 09:22:18 Waiting for src bucket item count to get to: 50000 curr count: 43368
2021/12/16 09:22:29 Waiting for dst bucket item count to get to: 50000 curr count: 34724
2021/12/16 09:22:34 Waiting for dst bucket item count to get to: 50000 curr count: 34724
2021/12/16 09:22:39 Waiting for dst bucket item count to get to: 50000 curr count: 34724
2021/12/16 09:22:44 Waiting for dst bucket item count to get to: 50000 curr count: 47486
2021/12/16 09:22:49 Waiting for dst bucket item count to get to: 50000 curr count: 47486
2021/12/16 09:22:49 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:22:49 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:22:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:22:54 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:22:59 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:23:04 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:23:09 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:23:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:23:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/16 09:23:19 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/16 09:23:19 No eventing-con process running
2021/12/16 09:23:19 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/16 09:23:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:23:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:23:31 Waiting for dst bucket item count to get to: 0 curr count: 38882
2021/12/16 09:23:36 Waiting for dst bucket item count to get to: 0 curr count: 38882
2021/12/16 09:23:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:23:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:23:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:23:48 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:23:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (189.94s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/16 09:23:58 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:23:58 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 09:23:58 No eventing-con process running
2021/12/16 09:23:58 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/16 09:23:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:24:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:24:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:24:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:24:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:24:16 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:24:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 09:24:22 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 09:24:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 09:24:32 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 09:24:37 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/16 09:24:37 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/16 09:24:37 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/16 09:24:37 Starting bucket ops to source bucket
2021/12/16 09:24:38 Removing node(s): [] from the cluster
2021/12/16 09:24:43 Rebalance progress: 12.5
2021/12/16 09:24:48 Rebalance progress: 12.5
2021/12/16 09:24:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:24:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:24:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 09:24:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:24:58 Rebalance progress: 12.5
2021/12/16 09:25:03 Rebalance progress: 12.5
2021/12/16 09:25:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:25:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:25:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:25:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:25:14 Rebalance progress: 12.5
2021/12/16 09:25:19 Rebalance progress: 12.5
2021/12/16 09:25:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:25:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:25:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:25:24 Waiting for src bucket item count to get to: 50000 curr count: 47177
2021/12/16 09:25:29 Waiting for src bucket item count to get to: 50000 curr count: 47177
2021/12/16 09:25:39 Waiting for dst bucket item count to get to: 50000 curr count: 46718
2021/12/16 09:25:44 Waiting for dst bucket item count to get to: 50000 curr count: 46718
2021/12/16 09:25:44 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:25:44 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:25:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:25:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:25:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:25:59 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:26:04 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:26:09 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:26:14 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/16 09:26:14 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/16 09:26:14 No eventing-con process running
2021/12/16 09:26:14 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/16 09:26:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:26:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:26:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:26:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:26:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:26:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:26:38 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:26:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (170.03s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/16 09:26:48 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:26:48 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 09:26:48 No eventing-con process running
2021/12/16 09:26:48 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/16 09:26:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:26:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:26:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:26:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:27:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:27:06 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:27:07 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/16 09:27:12 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 09:27:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 09:27:22 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 09:27:27 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/16 09:27:27 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/16 09:27:27 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:27:27 Starting bucket ops to source bucket
2021/12/16 09:27:28 Removing node(s): [] from the cluster
2021/12/16 09:27:33 Rebalance progress: 12.5
2021/12/16 09:27:38 Rebalance progress: 12.5
2021/12/16 09:27:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:27:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:27:48 Rebalance progress: 12.5
2021/12/16 09:27:53 Rebalance progress: 12.5
2021/12/16 09:27:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:27:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:27:58 Waiting for src bucket item count to get to: 50000 curr count: 25556
2021/12/16 09:28:03 Waiting for src bucket item count to get to: 50000 curr count: 36056
2021/12/16 09:28:08 Waiting for src bucket item count to get to: 50000 curr count: 36056
2021/12/16 09:28:13 Waiting for src bucket item count to get to: 50000 curr count: 48175
2021/12/16 09:28:18 Waiting for src bucket item count to get to: 50000 curr count: 48175
2021/12/16 09:28:28 Waiting for dst bucket item count to get to: 50000 curr count: 46056
2021/12/16 09:28:33 Waiting for dst bucket item count to get to: 50000 curr count: 46056
2021/12/16 09:28:33 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:28:33 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:28:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:28:38 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:28:43 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:28:49 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:28:54 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:28:59 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/16 09:28:59 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/16 09:28:59 No eventing-con process running
2021/12/16 09:28:59 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/16 09:29:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:29:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:29:10 Waiting for dst bucket item count to get to: 0 curr count: 25952
2021/12/16 09:29:15 Waiting for dst bucket item count to get to: 0 curr count: 25952
2021/12/16 09:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:29:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:29:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:29:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:29:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.60s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/16 09:29:32 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:29:33 Removing node(s): [] from the cluster
2021/12/16 09:29:38 Rebalance progress: 100
2021/12/16 09:29:43 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:29:43 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 09:29:43 No eventing-con process running
2021/12/16 09:29:43 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/16 09:29:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:29:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:29:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:29:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:29:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:29:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:30:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:30:02 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:30:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/16 09:30:08 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 09:30:13 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 09:30:18 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 09:30:23 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/16 09:30:23 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/16 09:30:23 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:30:23 Starting bucket ops to source bucket
2021/12/16 09:30:28 Rebalance progress: 12.5
2021/12/16 09:30:33 Rebalance progress: 12.5
2021/12/16 09:30:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:30:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:30:38 Waiting for src bucket item count to get to: 50000 curr count: 8308
2021/12/16 09:30:43 Waiting for src bucket item count to get to: 50000 curr count: 20691
2021/12/16 09:30:48 Waiting for src bucket item count to get to: 50000 curr count: 20691
2021/12/16 09:30:53 Waiting for src bucket item count to get to: 50000 curr count: 32075
2021/12/16 09:30:58 Waiting for src bucket item count to get to: 50000 curr count: 32075
2021/12/16 09:31:03 Waiting for src bucket item count to get to: 50000 curr count: 42510
2021/12/16 09:31:08 Waiting for src bucket item count to get to: 50000 curr count: 42510
2021/12/16 09:31:18 Waiting for dst bucket item count to get to: 50000 curr count: 38542
2021/12/16 09:31:23 Waiting for dst bucket item count to get to: 50000 curr count: 38542
2021/12/16 09:31:28 Waiting for dst bucket item count to get to: 50000 curr count: 38542
2021/12/16 09:31:28 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:31:28 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:31:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:31:33 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:38 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:43 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:48 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/16 09:31:58 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/16 09:31:58 No eventing-con process running
2021/12/16 09:31:58 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/16 09:32:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:32:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:32:10 Waiting for dst bucket item count to get to: 0 curr count: 14980
2021/12/16 09:32:15 Waiting for dst bucket item count to get to: 0 curr count: 14980
2021/12/16 09:32:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:32:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:32:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:32:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.74s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/16 09:32:37 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:32:37 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 09:32:37 No eventing-con process running
2021/12/16 09:32:37 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/16 09:32:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:32:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:32:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:32:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:32:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:32:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:32:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:32:56 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:32:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/16 09:33:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 09:33:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 09:33:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 09:33:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/16 09:33:17 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/16 09:33:17 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:33:17 Starting bucket ops to source bucket
2021/12/16 09:33:18 Removing node(s): [] from the cluster
2021/12/16 09:33:23 Rebalance progress: 12.5
2021/12/16 09:33:28 Rebalance progress: 12.5
2021/12/16 09:33:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:33:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:33:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:33:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:33:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:33:38 Rebalance progress: 12.5
2021/12/16 09:33:43 Rebalance progress: 12.5
2021/12/16 09:33:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:33:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:33:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:33:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:33:53 Rebalance progress: 12.5
2021/12/16 09:33:58 Rebalance progress: 12.5
2021/12/16 09:34:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:34:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:34:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:34:03 Waiting for src bucket item count to get to: 50000 curr count: 48246
2021/12/16 09:34:08 Waiting for src bucket item count to get to: 50000 curr count: 48246
2021/12/16 09:34:18 Waiting for dst bucket item count to get to: 50000 curr count: 46396
2021/12/16 09:34:23 Waiting for dst bucket item count to get to: 50000 curr count: 46396
2021/12/16 09:34:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/16 09:34:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/16 09:34:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:34:28 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:33 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:44 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:49 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:54 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/16 09:34:54 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/16 09:34:54 No eventing-con process running
2021/12/16 09:34:54 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/16 09:34:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:35:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:35:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:35:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:35:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:35:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:35:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/16 09:35:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.23s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/16 09:35:27 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:35:27 App: bucket_op_on_update got un-deployed
2021/12/16 09:35:27 No eventing-con process running
2021/12/16 09:35:27 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:35:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:35:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:35:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:35:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:35:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:35:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:35:46 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:35:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:35:52 Starting bucket ops to source bucket
2021/12/16 09:35:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:36:02 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:36:07 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:36:07 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:36:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:36:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:36:12 Rebalance progress: 12.5
2021/12/16 09:36:17 Rebalance progress: 12.5
2021/12/16 09:36:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:36:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:36:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:36:28 Rebalance progress: 12.5
2021/12/16 09:36:33 Rebalance progress: 12.5
2021/12/16 09:36:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:36:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 09:36:39 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:36:44 Rebalance progress: 12.5
2021/12/16 09:36:49 Rebalance progress: 12.5
2021/12/16 09:36:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:36:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:36:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:36:59 Rebalance progress: 12.5
2021/12/16 09:37:04 Rebalance progress: 12.5
2021/12/16 09:37:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:37:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:37:14 Rebalance progress: 12.5
2021/12/16 09:37:19 Rebalance progress: 12.5
2021/12/16 09:37:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:37:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 09:37:29 Rebalance progress: 12.5
2021/12/16 09:37:34 Rebalance progress: 12.5
2021/12/16 09:37:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:37:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:37:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:37:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:37:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:38:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:38:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:38:10 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:38:10 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:38:10 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:38:10 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:38:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:38:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:38:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10719
2021/12/16 09:38:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10719
2021/12/16 09:38:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:38:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:38:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:38:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10719
2021/12/16 09:38:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.98s)
=== RUN   TestKVForceFailover
2021/12/16 09:38:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:38:53 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:38:53 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:38:53 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:38:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:39:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:39:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:39:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:39:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:39:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:39:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:39:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:39:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:39:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:39:12 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:39:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:39:13 Starting bucket ops to source bucket
2021/12/16 09:39:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:39:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:39:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:39:33 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:39:33 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:39:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:39:33 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 09:39:53 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 09:39:53 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/16 09:39:59 Rebalance progress: 12.5
2021/12/16 09:40:04 Rebalance progress: 12.5
2021/12/16 09:40:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:40:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:40:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/16 09:40:14 Rebalance progress: 12.5
2021/12/16 09:40:19 Rebalance progress: 12.5
2021/12/16 09:40:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:40:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:40:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:40:24 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:40:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:40:54 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:40:54 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:40:54 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:40:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:41:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:41:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7034
2021/12/16 09:41:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:41:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:41:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:41:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7034
2021/12/16 09:41:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.35s)
=== RUN   TestKVHardFailoverRecovery
2021/12/16 09:41:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:41:28 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:41:28 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:41:28 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:41:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:41:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:41:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:41:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:41:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:41:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:41:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:41:46 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:41:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:41:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:41:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:42:02 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:42:07 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:42:07 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:42:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:42:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:42:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:42:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:42:13 Rebalance progress: 12.5
2021/12/16 09:42:18 Rebalance progress: 12.5
2021/12/16 09:42:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:42:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:42:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:42:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/16 09:43:23 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/16 09:43:28 Rebalance progress: 12.5
2021/12/16 09:43:33 Rebalance progress: 12.5
2021/12/16 09:43:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:43:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:43:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:43:43 Rebalance progress: 12.5
2021/12/16 09:43:48 Rebalance progress: 12.5
2021/12/16 09:43:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:43:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:43:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:43:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:24 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:44:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:44:24 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:44:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:44:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:44:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12092
2021/12/16 09:44:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:44:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:44:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:44:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12092
2021/12/16 09:44:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.68s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/16 09:44:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:44:57 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:44:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:44:57 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:44:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:45:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:45:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:45:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:45:16 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:45:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:45:22 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:45:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:45:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:45:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:45:37 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:45:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:45:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:45:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 09:45:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:45:43 Rebalance progress: 12.5
2021/12/16 09:45:48 Rebalance progress: 12.5
2021/12/16 09:45:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:45:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:45:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/16 09:45:58 Rebalance progress: 12.5
2021/12/16 09:46:03 Rebalance progress: 12.5
2021/12/16 09:46:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:46:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:46:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:46:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:46:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:46:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:46:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:46:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:46:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:46:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 133
2021/12/16 09:46:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 133
2021/12/16 09:46:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:46:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:47:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4577
2021/12/16 09:47:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.45s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/16 09:47:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:47:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:47:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:47:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:47:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:47:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:47:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:47:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:47:30 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:47:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:47:36 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:47:41 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:47:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:47:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:47:51 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:47:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:47:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:47:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:47:57 Rebalance progress: 12.5
2021/12/16 09:48:02 Rebalance progress: 12.5
2021/12/16 09:48:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:48:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:48:08 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:48:13 Rebalance progress: 12.5
2021/12/16 09:48:18 Rebalance progress: 12.5
2021/12/16 09:48:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:48:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:48:28 Rebalance progress: 12.5
2021/12/16 09:48:33 Rebalance progress: 12.5
2021/12/16 09:48:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:48:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:48:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:48:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:48:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:48:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:48:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:49:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:49:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:49:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:49:08 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:49:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:49:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:49:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:49:20 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 127
2021/12/16 09:49:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 127
2021/12/16 09:49:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:49:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:49:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:49:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6119
2021/12/16 09:49:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:49:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:49:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:49:42 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:49:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:49:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:49:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:49:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:49:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:49:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:49:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.71s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/16 09:50:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:50:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:50:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:50:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:50:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:50:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:50:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:50:24 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 09:50:24 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:50:24 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 09:50:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:50:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:38 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:50:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:50:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:50:39 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:50:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 09:50:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:50:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:50:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:51:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:51:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 09:51:05 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:51:10 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:51:15 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:51:15 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:51:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:51:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:51:21 Rebalance progress: 12.5
2021/12/16 09:51:26 Rebalance progress: 12.5
2021/12/16 09:51:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:51:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:51:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:51:36 Rebalance progress: 12.5
2021/12/16 09:51:41 Rebalance progress: 12.5
2021/12/16 09:51:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:51:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/16 09:51:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:51:51 Rebalance progress: 12.5
2021/12/16 09:51:56 Rebalance progress: 12.5
2021/12/16 09:52:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:52:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:52:07 Rebalance progress: 12.5
2021/12/16 09:52:12 Rebalance progress: 12.5
2021/12/16 09:52:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:52:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:52:22 Rebalance progress: 12.5
2021/12/16 09:52:27 Rebalance progress: 12.5
2021/12/16 09:52:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:52:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/16 09:52:37 Rebalance progress: 12.5
2021/12/16 09:52:42 Rebalance progress: 12.5
2021/12/16 09:52:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:52:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:52:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:52:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:52:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:53:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:53:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:53:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:53:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:53:17 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:53:17 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:53:17 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:53:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:53:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:53:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3817
2021/12/16 09:53:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3817
2021/12/16 09:53:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:53:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:53:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:53:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:53:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:53:50 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:53:55 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:00 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:05 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:10 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:15 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:15 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 09:54:15 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:54:15 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 09:54:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:54:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:54:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:54:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:54:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:54:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (273.33s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/16 09:54:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:54:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:54:39 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:54:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:54:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:54:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:54:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:54:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:54:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:54:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:54:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:54:57 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 09:54:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:54:57 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/16 09:54:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:55:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:55:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:55:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:55:11 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:55:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:55:12 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:55:13 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/16 09:55:18 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:55:23 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:55:28 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:55:33 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:55:33 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/16 09:55:38 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:55:43 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:55:48 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/16 09:55:48 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:55:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:55:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:55:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:55:54 Rebalance progress: 12.5
2021/12/16 09:55:59 Rebalance progress: 12.5
2021/12/16 09:56:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:56:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:56:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:56:10 Rebalance progress: 12.5
2021/12/16 09:56:15 Rebalance progress: 12.5
2021/12/16 09:56:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:56:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 09:56:25 Rebalance progress: 12.5
2021/12/16 09:56:30 Rebalance progress: 12.5
2021/12/16 09:56:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:56:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:56:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:56:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:56:45 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:56:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:56:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:57:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:57:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:57:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:57:05 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:57:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 09:57:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:57:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:57:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:57:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:57:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:57:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7638
2021/12/16 09:57:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:57:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 09:57:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:52 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/16 09:57:57 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/16 09:57:57 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:57:57 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/16 09:57:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:58:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:58:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:58:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:58:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:58:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (221.94s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/16 09:58:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 09:58:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 09:58:21 eventingSetup.go:46: No eventing-con process running
2021/12/16 09:58:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 09:58:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:58:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:58:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 09:58:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:58:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:58:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:58:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 09:58:39 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 09:58:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 09:58:45 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 09:58:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:58:55 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:59:00 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 09:59:00 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:59:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 09:59:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:59:21 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:59:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 09:59:37 Rebalance progress: 12.5
2021/12/16 09:59:42 Rebalance progress: 12.5
2021/12/16 09:59:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 09:59:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 09:59:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 09:59:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 09:59:52 Rebalance progress: 12.5
2021/12/16 09:59:57 Rebalance progress: 12.5
2021/12/16 10:00:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:00:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:00:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 10:00:07 Rebalance progress: 12.5
2021/12/16 10:00:12 Rebalance progress: 12.5
2021/12/16 10:00:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:00:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:00:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:00:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:00:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:00:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 10:00:48 eventingSetup.go:46: No eventing-con process running
2021/12/16 10:00:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 10:00:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:00:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:00:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:00:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:01:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9162
2021/12/16 10:01:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.42s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/16 10:01:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:01:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 10:01:16 eventingSetup.go:46: No eventing-con process running
2021/12/16 10:01:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/16 10:01:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:01:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:01:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:01:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:01:35 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:01:36 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 10:01:41 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/16 10:01:46 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 10:01:51 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 10:01:56 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/16 10:01:56 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:01:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:01:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 10:01:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 10:02:01 Rebalance progress: 12.5
2021/12/16 10:02:06 Rebalance progress: 12.5
2021/12/16 10:02:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:02:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:02:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 10:02:26 Rebalance progress: 12.5
2021/12/16 10:02:31 Rebalance progress: 12.5
2021/12/16 10:02:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:02:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/16 10:02:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/16 10:02:42 Rebalance progress: 12.5
2021/12/16 10:02:47 Rebalance progress: 12.5
2021/12/16 10:02:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:02:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/16 10:02:57 Rebalance progress: 12.5
2021/12/16 10:03:02 Rebalance progress: 12.5
2021/12/16 10:03:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:03:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:03:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:03:37 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 10:03:37 eventingSetup.go:46: No eventing-con process running
2021/12/16 10:03:37 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 10:03:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:03:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:03:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8579
2021/12/16 10:03:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:03:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:03:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:04:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6826
2021/12/16 10:04:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6826
2021/12/16 10:04:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (179.69s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/16 10:04:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/16 10:04:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/16 10:04:16 Rebalance progress: 100
2021/12/16 10:04:16 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:04:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:04:46 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/16 10:04:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/16 10:04:53 Starting bucket ops to source bucket
2021/12/16 10:04:53 Removing node(s): [] from the cluster
2021/12/16 10:04:58 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/16 10:05:03 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 10:05:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/16 10:05:08 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:05:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/16 10:05:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/16 10:05:13 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:23 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:28 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/16 10:05:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/16 10:05:38 eventingSetup.go:46: No eventing-con process running
2021/12/16 10:05:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/16 10:05:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:05:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:05:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/16 10:05:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/16 10:05:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/16 10:05:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1501
2021/12/16 10:05:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (110.73s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12109.586s

Rebalance tests took 3 hours, 21 minutes and 52 seconds.

Testing Succeeded


Version: versions-16.12.2021-02.25.cfg
Build Log: make-16.12.2021-02.25.log
Server Log: logs-16.12.2021-02.25.tar.gz

Finished