Building

Started building at 2021/12/13 18:53:53
Calculating base
Updating mirror
Basing run on 7.1.0-1899 c456ae0eb
Updating tree for run 13.12.2021-18.53
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at 888c51e, changes since last good build: none
kv_engine is at 9142cb0, changes since last good build: 
 9142cb07c Add mcbasher
ns_server is at 5bd87dc, changes since last good build: none
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at 2f97349, changes since last good build: none
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 3ae49c5, changes since last good build: 
 3ae49c5 MB-50023 : getLibraries endpoint should determine libraries list from metakv instead of in-memory store
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 4e1f64d3] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 98303e95] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/17
Total 4 (delta 3), reused 4 (delta 3)
[unstable 168dc944] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable a4da3ee6] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 5d302e48] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 9036e924] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Pulling in uncommitted change 166999 at refs/changes/99/166999/5
Total 56 (delta 47), reused 56 (delta 47)
[unstable 82277d36] MB-46304: Add functionality to disable eventing exposed javascript features
 Author: AnkitPrabhu 
 Date: Mon Dec 6 13:26:12 2021 +0530
 20 files changed, 217 insertions(+), 30 deletions(-)
Pulling in uncommitted change 167196 at refs/changes/96/167196/6
Total 67 (delta 57), reused 67 (delta 57)
[unstable a9e5a7d3] MB-49720: Store failover log in checkpoint blob
 Author: AnkitPrabhu 
 Date: Wed Dec 8 15:16:22 2021 +0530
 5 files changed, 147 insertions(+), 72 deletions(-)
Pulling in uncommitted change 167183 at refs/changes/83/167183/2
Total 5 (delta 2), reused 4 (delta 2)
[unstable 4019bc5] MB-46304: Throw runtime exception when curl feature is disabled
 Author: AnkitPrabhu 
 Date: Wed Dec 8 10:39:19 2021 +0530
 1 file changed, 9 insertions(+), 3 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/13 19:11:00
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/13 19:12:41 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/13 19:12:52 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 19:12:58 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/13 19:12:58 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 19:13:03 Removing node(s): [] from the cluster
2021/12/13 19:13:08 Rebalance progress: 0
2021/12/13 19:13:13 Rebalance progress: 0.9403935185185185
2021/12/13 19:13:18 Rebalance progress: 5.533854166666667
2021/12/13 19:13:23 Rebalance progress: 9.25925925925926
2021/12/13 19:13:28 Rebalance progress: 13.61762152777778
2021/12/13 19:13:33 Rebalance progress: 18.35575810185185
2021/12/13 19:13:38 Rebalance progress: 21.6652199074074
2021/12/13 19:13:43 Rebalance progress: 26.38527199074074
2021/12/13 19:13:48 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/13 19:13:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:14:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:14:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:14:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:14:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:15:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:15:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:15:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:15:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:16:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:16:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:16:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:16:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:17:09 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:17:24 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:17:39 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:17:54 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/13 19:18:09 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:18:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:18:15 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 19:18:20 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 19:18:25 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 19:18:30 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/13 19:18:30 App: TestAdvancedGetOps got deployed
2021/12/13 19:18:30 Starting bucket ops to source bucket
2021/12/13 19:18:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 19:18:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 19:18:45 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 19:18:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 19:18:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:18:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:18:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:18:50 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 19:18:55 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 19:19:00 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 19:19:06 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/13 19:19:06 App: TestAdvancedGetOps got un-deployed
2021/12/13 19:19:06 No eventing-con process running
2021/12/13 19:19:06 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/13 19:19:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:19:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:19:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:19:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:19:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:19:34 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:19:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (340.21s)
=== RUN   TestAdvancedInsertOps
2021/12/13 19:19:34 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:19:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:19:40 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:19:45 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:19:50 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:19:55 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:19:55 App: TestAdvancedInsertOps got deployed
2021/12/13 19:19:55 Starting bucket ops to source bucket
2021/12/13 19:20:01 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 19:20:01 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 19:20:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:20:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:20:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:20:06 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:20:11 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:20:16 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:20:21 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:20:21 App: TestAdvancedInsertOps got un-deployed
2021/12/13 19:20:21 No eventing-con process running
2021/12/13 19:20:21 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/13 19:20:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:20:33 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:20:38 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:20:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:20:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:20:50 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:20:55 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:20:55 Testing insert operation on source bucket
2021/12/13 19:20:55 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:20:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:21:01 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:21:06 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:21:11 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:21:16 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/13 19:21:16 App: TestAdvancedInsertOps got deployed
2021/12/13 19:21:16 Starting bucket ops to source bucket
2021/12/13 19:21:22 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 19:21:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 19:21:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:21:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:21:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:21:27 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:21:32 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:21:37 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:21:42 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/13 19:21:42 App: TestAdvancedInsertOps got un-deployed
2021/12/13 19:21:42 No eventing-con process running
2021/12/13 19:21:42 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/13 19:21:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:21:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/13 19:21:59 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:21:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:22:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:22:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (151.64s)
=== RUN   TestAdvancedUpsertOps
2021/12/13 19:22:06 Starting bucket ops to source bucket
2021/12/13 19:22:06 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:22:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:22:13 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:22:18 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:22:23 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:22:28 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:22:28 App: TestAdvancedUpsertOps got deployed
2021/12/13 19:22:28 Starting bucket ops to source bucket
2021/12/13 19:22:34 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:22:39 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:22:44 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:22:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:22:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:22:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:22:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:22:49 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:22:54 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:22:59 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:23:04 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:23:04 App: TestAdvancedUpsertOps got un-deployed
2021/12/13 19:23:04 No eventing-con process running
2021/12/13 19:23:04 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/13 19:23:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:23:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:23:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:23:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:23:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:23:22 Testing upsert operation on source bucket
2021/12/13 19:23:22 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:23:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:23:29 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:23:34 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:23:39 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:23:44 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/13 19:23:44 App: TestAdvancedUpsertOps got deployed
2021/12/13 19:23:44 Starting bucket ops to source bucket
2021/12/13 19:23:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:23:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:23:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:23:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:23:50 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:23:55 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:24:00 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:24:05 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/13 19:24:05 App: TestAdvancedUpsertOps got un-deployed
2021/12/13 19:24:05 No eventing-con process running
2021/12/13 19:24:05 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/13 19:24:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:24:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:24:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:24:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (133.09s)
=== RUN   TestAdvancedReplaceOps
2021/12/13 19:24:19 Starting bucket ops to source bucket
2021/12/13 19:24:20 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:24:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:24:26 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:24:31 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:24:36 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:24:41 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:24:41 App: TestAdvancedReplaceOps got deployed
2021/12/13 19:24:41 Starting bucket ops to source bucket
2021/12/13 19:24:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:24:52 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:24:57 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/13 19:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:24:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:24:57 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:24:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:25:02 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:25:07 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:25:12 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:25:17 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:25:17 App: TestAdvancedReplaceOps got un-deployed
2021/12/13 19:25:17 No eventing-con process running
2021/12/13 19:25:17 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/13 19:25:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:25:29 Waiting for dst bucket item count to get to: 0 curr count: 1011
2021/12/13 19:25:34 Waiting for dst bucket item count to get to: 0 curr count: 499
2021/12/13 19:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:25:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:25:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:25:41 Testing upsert operation on source bucket
2021/12/13 19:25:41 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:25:42 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:25:47 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:25:52 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:25:57 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:26:02 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/13 19:26:02 App: TestAdvancedReplaceOps got deployed
2021/12/13 19:26:02 Starting bucket ops to source bucket
2021/12/13 19:26:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:26:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:26:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:26:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:26:08 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:26:13 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:26:18 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:26:23 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/13 19:26:23 App: TestAdvancedReplaceOps got un-deployed
2021/12/13 19:26:23 No eventing-con process running
2021/12/13 19:26:23 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/13 19:26:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:26:35 Waiting for dst bucket item count to get to: 0 curr count: 347
2021/12/13 19:26:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:26:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:26:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (142.76s)
=== RUN   TestAdvancedDeleteOps
2021/12/13 19:26:42 Starting bucket ops to source bucket
2021/12/13 19:26:42 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:26:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:26:48 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:26:53 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:26:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:27:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:27:03 App: TestAdvancedDeleteOps got deployed
2021/12/13 19:27:03 Starting bucket ops to source bucket
2021/12/13 19:27:08 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:27:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:27:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:27:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:27:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:27:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:27:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:27:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:27:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:27:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:27:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/13 19:27:33 No eventing-con process running
2021/12/13 19:27:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/13 19:27:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:27:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:27:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:27:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:27:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:27:52 Testing delete operation on source bucket
2021/12/13 19:27:52 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:27:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/13 19:27:58 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:28:03 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:28:08 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:28:13 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/13 19:28:13 App: TestAdvancedDeleteOps got deployed
2021/12/13 19:28:13 Starting bucket ops to source bucket
2021/12/13 19:28:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:28:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:28:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:28:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:28:18 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:28:23 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:28:28 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:28:33 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/13 19:28:33 App: TestAdvancedDeleteOps got un-deployed
2021/12/13 19:28:33 No eventing-con process running
2021/12/13 19:28:33 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/13 19:28:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:28:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:28:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:28:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.74s)
=== RUN   TestEnoentAdvancedGet
2021/12/13 19:28:47 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:28:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/13 19:28:53 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:28:58 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:29:03 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:29:08 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:29:08 App: TestEnoentAdvancedGet got deployed
2021/12/13 19:29:08 Starting bucket ops to source bucket
2021/12/13 19:29:14 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 19:29:19 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 19:29:19 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 19:29:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:29:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:29:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:29:24 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:29:29 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:29:34 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:29:39 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:29:39 App: TestEnoentAdvancedGet got un-deployed
2021/12/13 19:29:39 No eventing-con process running
2021/12/13 19:29:39 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/13 19:29:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:29:51 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/13 19:29:56 Waiting for dst bucket item count to get to: 0 curr count: 82
2021/12/13 19:29:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:29:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:30:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:30:07 Testing get enoent operation on source bucket
2021/12/13 19:30:07 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:30:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/13 19:30:13 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:30:19 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:30:24 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:30:29 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/13 19:30:29 App: TestEnoentAdvancedGet got deployed
2021/12/13 19:30:29 Starting bucket ops to source bucket
2021/12/13 19:30:34 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 19:30:39 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 19:30:39 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 19:30:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:30:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:30:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:30:44 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:30:49 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:30:54 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:30:59 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/13 19:30:59 App: TestEnoentAdvancedGet got un-deployed
2021/12/13 19:30:59 No eventing-con process running
2021/12/13 19:30:59 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/13 19:31:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:31:11 Waiting for dst bucket item count to get to: 0 curr count: 174
2021/12/13 19:31:16 Waiting for dst bucket item count to get to: 0 curr count: 174
2021/12/13 19:31:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:31:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:31:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.08s)
=== RUN   TestEnoentAdvancedReplace
2021/12/13 19:31:22 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:31:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:31:29 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:31:34 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:31:39 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:31:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:31:44 App: TestEnoentAdvancedReplace got deployed
2021/12/13 19:31:44 Starting bucket ops to source bucket
2021/12/13 19:31:49 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 19:31:54 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 19:31:54 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 19:31:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:31:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:31:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:31:59 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:32:04 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:32:09 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:32:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:32:14 App: TestEnoentAdvancedReplace got un-deployed
2021/12/13 19:32:14 No eventing-con process running
2021/12/13 19:32:14 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/13 19:32:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:32:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:32:37 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:32:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:32:37 Testing get enoent operation on source bucket
2021/12/13 19:32:37 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:32:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:32:44 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:32:49 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:32:54 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:32:59 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/13 19:32:59 App: TestEnoentAdvancedReplace got deployed
2021/12/13 19:32:59 Starting bucket ops to source bucket
2021/12/13 19:33:04 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 19:33:09 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 19:33:09 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 19:33:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:33:09 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:33:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:33:14 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:33:19 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:33:24 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:33:29 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/13 19:33:29 App: TestEnoentAdvancedReplace got un-deployed
2021/12/13 19:33:29 No eventing-con process running
2021/12/13 19:33:29 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/13 19:33:31 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:33:41 Waiting for dst bucket item count to get to: 0 curr count: 175
2021/12/13 19:33:46 Waiting for dst bucket item count to get to: 0 curr count: 175
2021/12/13 19:33:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:33:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:33:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.00s)
=== RUN   TestEnoentAdvancedDelete
2021/12/13 19:33:52 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:33:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 19:33:59 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:34:04 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:34:09 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:34:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:34:14 App: TestEnoentAdvancedDelete got deployed
2021/12/13 19:34:14 Starting bucket ops to source bucket
2021/12/13 19:34:19 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 19:34:24 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 19:34:24 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 19:34:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:34:24 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:34:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:34:29 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:34:34 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:34:39 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:34:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:34:44 App: TestEnoentAdvancedDelete got un-deployed
2021/12/13 19:34:44 No eventing-con process running
2021/12/13 19:34:44 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/13 19:34:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:34:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:34:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:34:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:35:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:35:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:35:07 Testing get enoent operation on source bucket
2021/12/13 19:35:07 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:35:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 19:35:14 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:35:19 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:35:24 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:35:29 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/13 19:35:29 App: TestEnoentAdvancedDelete got deployed
2021/12/13 19:35:29 Starting bucket ops to source bucket
2021/12/13 19:35:34 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/13 19:35:39 Waiting for dst bucket item count to get to: 200 curr count: 100
2021/12/13 19:35:39 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/13 19:35:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:35:39 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:35:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:35:44 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:35:49 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:35:54 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:35:59 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/13 19:35:59 App: TestEnoentAdvancedDelete got un-deployed
2021/12/13 19:35:59 No eventing-con process running
2021/12/13 19:35:59 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/13 19:36:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:36:11 Waiting for dst bucket item count to get to: 0 curr count: 173
2021/12/13 19:36:16 Waiting for dst bucket item count to get to: 0 curr count: 173
2021/12/13 19:36:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:36:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:36:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (149.95s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/13 19:36:22 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:36:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/13 19:36:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:36:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:36:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:36:44 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:36:44 App: TestAdvancedInsertKeyExist got deployed
2021/12/13 19:36:44 Starting bucket ops to source bucket
2021/12/13 19:36:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:36:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:36:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:36:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:36:49 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:36:54 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:36:59 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:37:04 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:37:04 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/13 19:37:04 No eventing-con process running
2021/12/13 19:37:04 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/13 19:37:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:37:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:37:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:37:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:37:18 Pumping document in the destination bucket
2021/12/13 19:37:18 Starting bucket ops to source bucket
2021/12/13 19:37:18 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:37:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/13 19:37:24 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:37:29 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:37:34 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:37:39 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/13 19:37:39 App: TestAdvancedInsertKeyExist got deployed
2021/12/13 19:37:39 Starting bucket ops to source bucket
2021/12/13 19:37:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:37:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:37:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:37:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:37:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:37:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:37:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:38:00 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:38:05 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:38:10 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:38:15 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/13 19:38:15 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/13 19:38:15 No eventing-con process running
2021/12/13 19:38:15 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/13 19:38:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:38:26 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:38:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:38:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.71s)
=== RUN   TestExipryGet
2021/12/13 19:38:33 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:38:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/13 19:38:39 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:38:44 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:38:49 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:38:54 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:38:54 App: TestExipryGet got deployed
2021/12/13 19:38:54 Starting bucket ops to source bucket
2021/12/13 19:39:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:39:45 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:39:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:39:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:39:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:39:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:39:55 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:40:00 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:40:05 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:40:05 App: TestExipryGet got un-deployed
2021/12/13 19:40:05 No eventing-con process running
2021/12/13 19:40:05 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/13 19:40:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:40:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:40:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:40:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:40:19 Pumping document in the destination bucket for expiry get
2021/12/13 19:40:19 Starting bucket ops to source bucket
2021/12/13 19:40:19 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:40:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/13 19:40:25 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:40:30 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:40:35 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:40:40 Waiting for app: TestExipryGet to get deployed
2021/12/13 19:40:40 App: TestExipryGet got deployed
2021/12/13 19:40:40 Starting bucket ops to source bucket
2021/12/13 19:41:25 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:41:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:41:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:41:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:41:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:41:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:41:35 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:41:40 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:41:45 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:41:50 Waiting for app: TestExipryGet to get un-deployed
2021/12/13 19:41:50 App: TestExipryGet got un-deployed
2021/12/13 19:41:51 No eventing-con process running
2021/12/13 19:41:51 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/13 19:41:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:42:02 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:42:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:42:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:42:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:42:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (220.98s)
=== RUN   TestExipryInsert
2021/12/13 19:42:14 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:42:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/13 19:42:20 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:42:25 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:42:30 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:42:35 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:42:35 App: TestExipryInsert got deployed
2021/12/13 19:42:35 Starting bucket ops to source bucket
2021/12/13 19:43:20 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:43:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:43:20 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:43:20 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:43:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:43:26 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:43:31 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:43:36 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:43:41 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:43:41 App: TestExipryInsert got un-deployed
2021/12/13 19:43:41 No eventing-con process running
2021/12/13 19:43:41 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/13 19:43:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:43:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:43:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:43:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:43:55 Testing on the destination bucket for expiry insert
2021/12/13 19:43:55 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:43:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/13 19:44:01 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:44:06 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:44:11 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:44:16 Waiting for app: TestExipryInsert to get deployed
2021/12/13 19:44:16 App: TestExipryInsert got deployed
2021/12/13 19:44:16 Starting bucket ops to source bucket
2021/12/13 19:45:01 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:45:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:45:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:45:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:45:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:45:06 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:45:11 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:45:16 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:45:21 Waiting for app: TestExipryInsert to get un-deployed
2021/12/13 19:45:21 App: TestExipryInsert got un-deployed
2021/12/13 19:45:21 No eventing-con process running
2021/12/13 19:45:21 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/13 19:45:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:45:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:45:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:45:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:45:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (210.79s)
=== RUN   TestExipryReplace
2021/12/13 19:45:45 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:45:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:45:51 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:45:56 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:46:01 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:46:06 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:46:06 App: TestExipryReplace got deployed
2021/12/13 19:46:06 Starting bucket ops to source bucket
2021/12/13 19:46:51 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:46:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:46:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:46:52 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:46:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:46:57 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:47:02 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:47:07 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:47:12 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:47:12 App: TestExipryReplace got un-deployed
2021/12/13 19:47:12 No eventing-con process running
2021/12/13 19:47:12 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/13 19:47:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:47:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:47:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:47:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:47:26 Testing on the destination bucket for expiry insert
2021/12/13 19:47:26 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:47:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:47:32 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:47:37 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:47:42 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:47:47 Waiting for app: TestExipryReplace to get deployed
2021/12/13 19:47:47 App: TestExipryReplace got deployed
2021/12/13 19:47:47 Starting bucket ops to source bucket
2021/12/13 19:48:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:48:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:48:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:48:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:48:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:48:37 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:48:42 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:48:47 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:48:52 Waiting for app: TestExipryReplace to get un-deployed
2021/12/13 19:48:52 App: TestExipryReplace got un-deployed
2021/12/13 19:48:52 No eventing-con process running
2021/12/13 19:48:52 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/13 19:48:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:49:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:49:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:49:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:49:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:49:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (210.83s)
=== RUN   TestExipryUpsert
2021/12/13 19:49:16 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:49:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/13 19:49:22 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:49:27 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:49:32 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:49:37 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:49:37 App: TestExipryUpsert got deployed
2021/12/13 19:49:37 Starting bucket ops to source bucket
2021/12/13 19:50:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:50:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:50:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:50:22 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:50:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:50:27 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:50:32 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:50:37 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:50:42 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:50:42 App: TestExipryUpsert got un-deployed
2021/12/13 19:50:42 No eventing-con process running
2021/12/13 19:50:42 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/13 19:50:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:50:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:50:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:50:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:50:56 Testing on the destination bucket for expiry insert
2021/12/13 19:50:56 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:50:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/13 19:51:02 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:51:07 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:51:12 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:51:17 Waiting for app: TestExipryUpsert to get deployed
2021/12/13 19:51:17 App: TestExipryUpsert got deployed
2021/12/13 19:51:17 Starting bucket ops to source bucket
2021/12/13 19:52:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:52:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:52:13 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:52:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:52:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:52:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:52:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:52:18 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:52:23 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:52:28 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:52:33 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/13 19:52:33 App: TestExipryUpsert got un-deployed
2021/12/13 19:52:33 No eventing-con process running
2021/12/13 19:52:33 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/13 19:52:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:52:45 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:52:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:52:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (215.64s)
=== RUN   TestCasReplace
2021/12/13 19:52:51 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:52:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:52:58 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:53:03 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:53:08 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:53:13 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:53:13 App: TestCasReplace got deployed
2021/12/13 19:53:13 Starting bucket ops to source bucket
2021/12/13 19:53:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:53:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:53:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:53:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:53:18 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:53:23 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:53:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:53:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:53:33 App: TestCasReplace got un-deployed
2021/12/13 19:53:33 No eventing-con process running
2021/12/13 19:53:33 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/13 19:53:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:53:45 Waiting for dst bucket item count to get to: 0 curr count: 4
2021/12/13 19:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:53:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:53:51 Testing on the destination bucket for upsert with cas
2021/12/13 19:53:51 Starting bucket ops to source bucket
2021/12/13 19:53:52 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:53:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/13 19:53:58 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:54:03 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:54:08 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:54:13 Waiting for app: TestCasReplace to get deployed
2021/12/13 19:54:13 App: TestCasReplace got deployed
2021/12/13 19:54:13 Starting bucket ops to source bucket
2021/12/13 19:54:18 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:54:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:54:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:54:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:54:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:54:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:54:28 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:54:33 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:54:38 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:54:43 Waiting for app: TestCasReplace to get un-deployed
2021/12/13 19:54:43 App: TestCasReplace got un-deployed
2021/12/13 19:54:43 No eventing-con process running
2021/12/13 19:54:43 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/13 19:54:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:54:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:54:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:55:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.27s)
=== RUN   TestCasDelete
2021/12/13 19:55:02 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:55:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 19:55:08 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:55:13 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:55:18 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:55:23 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:55:23 App: TestCasDelete got deployed
2021/12/13 19:55:23 Starting bucket ops to source bucket
2021/12/13 19:55:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:55:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:55:23 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:55:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:55:28 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:55:33 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:55:38 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:55:43 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:55:43 App: TestCasDelete got un-deployed
2021/12/13 19:55:43 No eventing-con process running
2021/12/13 19:55:43 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/13 19:55:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:55:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:55:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:55:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:55:57 Testing on the destination bucket for delete with cas
2021/12/13 19:55:57 Starting bucket ops to source bucket
2021/12/13 19:55:57 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:55:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 19:56:03 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:56:08 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:56:13 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:56:18 Waiting for app: TestCasDelete to get deployed
2021/12/13 19:56:18 App: TestCasDelete got deployed
2021/12/13 19:56:18 Starting bucket ops to source bucket
2021/12/13 19:56:24 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 19:56:29 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 19:56:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:56:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:56:29 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 19:56:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:56:34 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:56:39 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:56:44 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:56:49 Waiting for app: TestCasDelete to get un-deployed
2021/12/13 19:56:49 App: TestCasDelete got un-deployed
2021/12/13 19:56:49 No eventing-con process running
2021/12/13 19:56:49 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/13 19:56:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:57:01 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/13 19:57:06 Waiting for dst bucket item count to get to: 0 curr count: 84
2021/12/13 19:57:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:57:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:57:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (130.81s)
=== RUN   TestCountersIncrement
2021/12/13 19:57:12 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 19:57:18 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 19:57:19 Removing node(s): [] from the cluster
2021/12/13 19:57:24 Rebalance progress: 100
2021/12/13 19:57:24 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:57:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/13 19:57:30 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:57:35 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:57:40 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:57:45 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:57:45 App: TestCountersIncrement got deployed
2021/12/13 19:57:45 Starting bucket ops to source bucket
2021/12/13 19:57:51 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 19:57:56 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/13 19:57:56 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 19:57:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:57:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:58:01 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:58:06 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:58:11 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:58:16 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:58:16 App: TestCountersIncrement got un-deployed
2021/12/13 19:58:16 No eventing-con process running
2021/12/13 19:58:16 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/13 19:58:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:58:28 Waiting for dst bucket item count to get to: 0 curr count: 728
2021/12/13 19:58:33 Waiting for dst bucket item count to get to: 0 curr count: 215
2021/12/13 19:58:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:58:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:58:45 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/13 19:58:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:58:45 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 19:58:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/13 19:58:51 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:58:56 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:59:01 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:59:06 Waiting for app: TestCountersIncrement to get deployed
2021/12/13 19:59:06 App: TestCountersIncrement got deployed
2021/12/13 19:59:06 Starting bucket ops to source bucket
2021/12/13 19:59:12 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 19:59:17 Waiting for dst bucket item count to get to: 2048 curr count: 1941
2021/12/13 19:59:22 Waiting for dst bucket item count to get to: 2048 curr count: 1941
2021/12/13 19:59:22 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 19:59:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 19:59:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 19:59:27 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:59:32 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:59:38 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:59:43 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/13 19:59:43 App: TestCountersIncrement got un-deployed
2021/12/13 19:59:43 No eventing-con process running
2021/12/13 19:59:43 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/13 19:59:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 19:59:54 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/13 19:59:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 19:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:00:06 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 20:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:00:06 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/13 20:00:11 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.83s)
=== RUN   TestCountersDecrement
2021/12/13 20:00:11 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 20:00:11 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 20:00:12 Removing node(s): [] from the cluster
2021/12/13 20:00:17 Rebalance progress: 100
2021/12/13 20:00:17 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:00:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/13 20:00:23 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:00:28 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:00:33 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:00:38 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:00:38 App: TestCountersDecrement got deployed
2021/12/13 20:00:38 Starting bucket ops to source bucket
2021/12/13 20:00:44 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/13 20:00:49 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/13 20:00:54 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/13 20:00:54 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/13 20:00:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:00:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:00:59 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:01:04 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:01:09 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:01:14 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:01:14 App: TestCountersDecrement got un-deployed
2021/12/13 20:01:14 No eventing-con process running
2021/12/13 20:01:14 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/13 20:01:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:01:26 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/13 20:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:01:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:01:38 Waiting for dst bucket item count to get to: 0 curr count: 664
2021/12/13 20:01:43 Waiting for dst bucket item count to get to: 0 curr count: 152
2021/12/13 20:01:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:01:43 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:01:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/13 20:01:49 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:01:54 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:01:59 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:02:04 Waiting for app: TestCountersDecrement to get deployed
2021/12/13 20:02:04 App: TestCountersDecrement got deployed
2021/12/13 20:02:04 Starting bucket ops to source bucket
2021/12/13 20:02:10 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/13 20:02:15 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/13 20:02:15 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/13 20:02:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:02:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:02:20 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:02:25 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:02:30 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:02:36 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/13 20:02:36 App: TestCountersDecrement got un-deployed
2021/12/13 20:02:36 No eventing-con process running
2021/12/13 20:02:36 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/13 20:02:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:02:47 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/13 20:02:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:02:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:02:59 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 20:02:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:02:59 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/13 20:03:04 Rebalance progress: 100
--- PASS: TestCountersDecrement (172.97s)
=== RUN   TestCacheGet
2021/12/13 20:03:04 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:03:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/13 20:03:10 Waiting for app: TestCacheGet to get deployed
2021/12/13 20:03:15 Waiting for app: TestCacheGet to get deployed
2021/12/13 20:03:20 Waiting for app: TestCacheGet to get deployed
2021/12/13 20:03:25 Waiting for app: TestCacheGet to get deployed
2021/12/13 20:03:25 App: TestCacheGet got deployed
2021/12/13 20:03:25 Starting bucket ops to source bucket
2021/12/13 20:03:31 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/13 20:03:36 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/13 20:03:41 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/13 20:03:46 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/13 20:03:51 Waiting for dst bucket item count to get to: 20 curr count: 12
2021/12/13 20:03:51 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/13 20:03:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:03:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:03:56 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 20:04:01 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 20:04:06 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 20:04:11 Waiting for app: TestCacheGet to get un-deployed
2021/12/13 20:04:11 App: TestCacheGet got un-deployed
2021/12/13 20:04:11 No eventing-con process running
2021/12/13 20:04:11 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/13 20:04:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:04:23 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/13 20:04:28 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/13 20:04:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:04:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:04:39 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/13 20:04:44 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/13 20:04:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (100.28s)
=== RUN   TestCacheRYOW
2021/12/13 20:04:44 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:04:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/13 20:04:51 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 20:04:56 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 20:05:01 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 20:05:06 Waiting for app: TestCacheRYOW to get deployed
2021/12/13 20:05:06 App: TestCacheRYOW got deployed
2021/12/13 20:05:06 Starting bucket ops to source bucket
2021/12/13 20:05:11 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/13 20:05:11 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/13 20:05:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:05:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:05:16 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 20:05:21 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 20:05:26 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 20:05:31 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/13 20:05:31 App: TestCacheRYOW got un-deployed
2021/12/13 20:05:31 No eventing-con process running
2021/12/13 20:05:31 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/13 20:05:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:05:43 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 20:05:48 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 20:05:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:05:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:06:00 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 20:06:05 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 20:06:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (80.20s)
=== RUN   TestCacheExpiry
2021/12/13 20:06:05 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:06:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/13 20:06:11 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 20:06:16 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 20:06:21 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 20:06:26 Waiting for app: TestCacheExpiry to get deployed
2021/12/13 20:06:26 App: TestCacheExpiry got deployed
2021/12/13 20:06:26 Starting bucket ops to source bucket
2021/12/13 20:06:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:06:36 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 20:06:41 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 20:06:46 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 20:06:51 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 20:06:51 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:06:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:06:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:06:56 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 20:07:01 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 20:07:06 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 20:07:11 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/13 20:07:11 App: TestCacheExpiry got un-deployed
2021/12/13 20:07:11 No eventing-con process running
2021/12/13 20:07:11 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/13 20:07:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:07:23 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 20:07:28 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 20:07:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:07:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:07:40 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/13 20:07:45 Waiting for dst bucket item count to get to: 0 curr count: 54
2021/12/13 20:07:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.18s)
=== RUN   TestCacheExpiryCustom
2021/12/13 20:07:45 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:07:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/13 20:07:51 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 20:07:56 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 20:08:01 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 20:08:06 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/13 20:08:06 App: TestCacheExpiryCustom got deployed
2021/12/13 20:08:06 Starting bucket ops to source bucket
2021/12/13 20:08:11 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:08:16 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 20:08:21 Waiting for dst bucket item count to get to: 100 curr count: 19
2021/12/13 20:08:26 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 20:08:31 Waiting for dst bucket item count to get to: 100 curr count: 94
2021/12/13 20:08:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:08:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:08:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:08:36 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 20:08:41 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 20:08:46 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 20:08:51 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/13 20:08:51 App: TestCacheExpiryCustom got un-deployed
2021/12/13 20:08:51 No eventing-con process running
2021/12/13 20:08:52 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/13 20:08:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:09:03 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 20:09:08 Waiting for dst bucket item count to get to: 0 curr count: 25
2021/12/13 20:09:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:09:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:09:20 Waiting for dst bucket item count to get to: 0 curr count: 62
2021/12/13 20:09:25 Waiting for dst bucket item count to get to: 0 curr count: 62
2021/12/13 20:09:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.22s)
=== RUN   TestCacheOverflowCustom
2021/12/13 20:09:25 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:09:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/13 20:09:31 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 20:09:36 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 20:09:41 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 20:09:46 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/13 20:09:46 App: TestCacheOverflowCustom got deployed
2021/12/13 20:09:46 Starting bucket ops to source bucket
2021/12/13 20:09:51 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/13 20:09:57 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/13 20:10:02 Waiting for dst bucket item count to get to: 35 curr count: 10
2021/12/13 20:10:07 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/13 20:10:12 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/13 20:10:12 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/13 20:10:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:10:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:10:17 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 20:10:22 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 20:10:27 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 20:10:32 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/13 20:10:32 App: TestCacheOverflowCustom got un-deployed
2021/12/13 20:10:32 No eventing-con process running
2021/12/13 20:10:32 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/13 20:10:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:10:44 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/13 20:10:49 Waiting for dst bucket item count to get to: 0 curr count: 3
2021/12/13 20:10:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:10:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:11:00 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/13 20:11:05 Waiting for dst bucket item count to get to: 0 curr count: 26
2021/12/13 20:11:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (100.41s)
=== RUN   TestStrictMode
2021/12/13 20:11:05 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:11:07 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.07s)
=== RUN   TestDataTypes
2021/12/13 20:11:07 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:11:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/13 20:11:13 Waiting for app: TestDataTypes to get deployed
2021/12/13 20:11:18 Waiting for app: TestDataTypes to get deployed
2021/12/13 20:11:23 Waiting for app: TestDataTypes to get deployed
2021/12/13 20:11:28 Waiting for app: TestDataTypes to get deployed
2021/12/13 20:11:28 App: TestDataTypes got deployed
2021/12/13 20:11:28 Starting bucket ops to source bucket
2021/12/13 20:11:33 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/13 20:11:38 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/13 20:11:43 Waiting for dst bucket item count to get to: 18 curr count: 10
2021/12/13 20:11:43 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/13 20:11:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:11:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:11:48 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 20:11:53 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 20:11:58 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 20:12:03 Waiting for app: TestDataTypes to get un-deployed
2021/12/13 20:12:03 App: TestDataTypes got un-deployed
2021/12/13 20:12:03 No eventing-con process running
2021/12/13 20:12:03 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/13 20:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:12:15 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 20:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:12:27 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/13 20:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (80.08s)
=== RUN   TestEnoentGet
2021/12/13 20:12:27 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:12:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/13 20:12:33 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:12:38 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:12:43 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:12:48 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:12:48 App: TestEnoentGet got deployed
2021/12/13 20:12:48 Starting bucket ops to source bucket
2021/12/13 20:12:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:12:58 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 20:13:03 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 20:13:03 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:13:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:13:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:13:08 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:13:13 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:13:18 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:13:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:13:23 App: TestEnoentGet got un-deployed
2021/12/13 20:13:23 No eventing-con process running
2021/12/13 20:13:23 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/13 20:13:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:13:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:13:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:13:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:13:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:13:47 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:13:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/13 20:13:53 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:13:58 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:14:03 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:14:08 Waiting for app: TestEnoentGet to get deployed
2021/12/13 20:14:08 App: TestEnoentGet got deployed
2021/12/13 20:14:08 Starting bucket ops to source bucket
2021/12/13 20:14:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:14:18 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 20:14:18 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:14:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:14:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:14:23 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:14:28 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:14:33 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:14:38 Waiting for app: TestEnoentGet to get un-deployed
2021/12/13 20:14:38 App: TestEnoentGet got un-deployed
2021/12/13 20:14:38 No eventing-con process running
2021/12/13 20:14:38 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/13 20:14:40 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:14:50 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/13 20:14:55 Waiting for dst bucket item count to get to: 0 curr count: 67
2021/12/13 20:14:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:14:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:15:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.36s)
=== RUN   TestEnoentDelete
2021/12/13 20:15:07 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:15:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 20:15:13 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:15:18 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:15:23 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:15:28 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:15:28 App: TestEnoentDelete got deployed
2021/12/13 20:15:28 Starting bucket ops to source bucket
2021/12/13 20:15:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:15:38 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 20:15:38 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:15:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:15:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:15:43 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:15:48 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:15:53 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:15:59 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:15:59 App: TestEnoentDelete got un-deployed
2021/12/13 20:15:59 No eventing-con process running
2021/12/13 20:15:59 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/13 20:16:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:16:10 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/13 20:16:15 Waiting for dst bucket item count to get to: 0 curr count: 76
2021/12/13 20:16:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:16:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:16:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:16:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:16:27 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:16:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 20:16:33 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:16:38 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:16:43 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:16:48 Waiting for app: TestEnoentDelete to get deployed
2021/12/13 20:16:48 App: TestEnoentDelete got deployed
2021/12/13 20:16:48 Starting bucket ops to source bucket
2021/12/13 20:16:53 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:16:58 Waiting for dst bucket item count to get to: 100 curr count: 50
2021/12/13 20:16:58 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:16:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:16:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:17:04 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:17:09 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:17:14 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:17:19 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/13 20:17:19 App: TestEnoentDelete got un-deployed
2021/12/13 20:17:19 No eventing-con process running
2021/12/13 20:17:19 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/13 20:17:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:17:30 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/13 20:17:35 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/13 20:17:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:17:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:17:47 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:17:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.27s)
=== RUN   TestError
2021/12/13 20:17:57 Waiting for app: TestError to get un-deployed
2021/12/13 20:17:57 App: TestError got un-deployed
2021/12/13 20:17:57 No eventing-con process running
2021/12/13 20:17:57 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/13 20:17:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:18:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:18:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:18:11 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:18:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/13 20:18:17 Waiting for app: TestError to get deployed
2021/12/13 20:18:22 Waiting for app: TestError to get deployed
2021/12/13 20:18:27 Waiting for app: TestError to get deployed
2021/12/13 20:18:32 Waiting for app: TestError to get deployed
2021/12/13 20:18:32 App: TestError got deployed
2021/12/13 20:18:32 Starting bucket ops to source bucket
2021/12/13 20:18:37 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/13 20:18:42 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/13 20:18:42 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/13 20:18:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:18:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:18:47 Waiting for app: TestError to get un-deployed
2021/12/13 20:18:53 Waiting for app: TestError to get un-deployed
2021/12/13 20:18:58 Waiting for app: TestError to get un-deployed
2021/12/13 20:19:03 Waiting for app: TestError to get un-deployed
2021/12/13 20:19:03 App: TestError got un-deployed
2021/12/13 20:19:03 No eventing-con process running
2021/12/13 20:19:03 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/13 20:19:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:19:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:19:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:19:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:19:26 Waiting for dst bucket item count to get to: 0 curr count: 300
2021/12/13 20:19:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (98.86s)
=== RUN   TestCRLF
2021/12/13 20:19:36 Waiting for app: TestCRLF to get un-deployed
2021/12/13 20:19:36 App: TestCRLF got un-deployed
2021/12/13 20:19:36 No eventing-con process running
2021/12/13 20:19:36 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/13 20:19:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:19:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:19:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:19:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:19:50 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:19:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/13 20:19:56 Waiting for app: TestCRLF to get deployed
2021/12/13 20:20:01 Waiting for app: TestCRLF to get deployed
2021/12/13 20:20:06 Waiting for app: TestCRLF to get deployed
2021/12/13 20:20:11 Waiting for app: TestCRLF to get deployed
2021/12/13 20:20:11 App: TestCRLF got deployed
2021/12/13 20:20:11 Starting bucket ops to source bucket
2021/12/13 20:20:16 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:20:21 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 20:20:21 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 20:20:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:20:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:20:26 Waiting for app: TestCRLF to get un-deployed
2021/12/13 20:20:31 Waiting for app: TestCRLF to get un-deployed
2021/12/13 20:20:36 Waiting for app: TestCRLF to get un-deployed
2021/12/13 20:20:41 Waiting for app: TestCRLF to get un-deployed
2021/12/13 20:20:41 App: TestCRLF got un-deployed
2021/12/13 20:20:41 No eventing-con process running
2021/12/13 20:20:41 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/13 20:20:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:20:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 20:20:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/13 20:20:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:21:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:21:10 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/13 20:21:15 Waiting for dst bucket item count to get to: 0 curr count: 61
2021/12/13 20:21:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (108.86s)
=== RUN   TestImportExport
2021/12/13 20:21:25 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:21:25 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 20:21:25 No eventing-con process running
2021/12/13 20:21:25 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/13 20:21:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:21:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:21:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:21:44 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:21:44 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 20:21:44 No eventing-con process running
2021/12/13 20:21:44 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/13 20:21:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:21:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:21:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:21:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:21:57 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:21:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 20:21:58 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:22:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 20:22:05 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 20:22:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 20:22:15 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 20:22:20 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/13 20:22:20 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/13 20:22:25 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 20:22:30 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 20:22:35 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/13 20:22:35 App: TestImportExport_bucket_op_on_update got deployed
2021/12/13 20:22:35 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:22:40 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:22:45 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:22:50 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:22:55 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:22:55 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 20:22:55 No eventing-con process running
2021/12/13 20:22:55 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/13 20:22:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:23:00 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:23:05 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:23:10 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:23:15 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:23:15 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 20:23:15 No eventing-con process running
2021/12/13 20:23:15 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/13 20:23:32 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/13 20:23:32 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/13 20:23:32 No eventing-con process running
2021/12/13 20:23:32 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/13 20:23:34 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:23:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:23:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:23:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:23:51 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/13 20:23:51 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/13 20:23:51 No eventing-con process running
2021/12/13 20:23:51 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/13 20:23:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:23:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:24:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:24:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.56s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/13 20:24:15 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:24:15 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:24:15 No eventing-con process running
2021/12/13 20:24:15 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/13 20:24:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:24:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:24:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:24:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:24:28 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:24:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:24:29 Starting bucket ops to source bucket
2021/12/13 20:24:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:24:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:24:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:24:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:24:57 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 20:24:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:25:02 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 20:25:02 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 20:25:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:25:03 Undeploying app: bucket_op_on_update
2021/12/13 20:25:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:25:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:25:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:25:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:25:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:25:23 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:25:23 No eventing-con process running
2021/12/13 20:25:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:25:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:26:07 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:26:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:26:08 Starting bucket ops to source bucket
2021/12/13 20:26:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:26:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:26:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:26:31 Waiting for dst bucket item count to get to: 5000 curr count: 315
2021/12/13 20:26:36 Waiting for dst bucket item count to get to: 5000 curr count: 2815
2021/12/13 20:26:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:26:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 20:26:41 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 20:26:41 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:26:41 Undeploying app: bucket_op_on_update
2021/12/13 20:26:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:26:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:26:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:26:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:27:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:27:01 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:27:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:27:01 No eventing-con process running
2021/12/13 20:27:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:27:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:27:45 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:27:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:27:46 Starting bucket ops to source bucket
2021/12/13 20:27:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:27:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:28:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:28:09 Waiting for dst bucket item count to get to: 5000 curr count: 1855
2021/12/13 20:28:14 Waiting for dst bucket item count to get to: 5000 curr count: 1855
2021/12/13 20:28:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:28:19 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 20:28:19 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 20:28:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:28:19 Undeploying app: bucket_op_on_update
2021/12/13 20:28:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:28:24 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:28:29 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:28:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:28:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:28:39 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:28:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:28:39 No eventing-con process running
2021/12/13 20:28:41 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:28:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:29:24 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:29:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:29:25 Starting bucket ops to source bucket
2021/12/13 20:29:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:29:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:29:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:29:48 Waiting for dst bucket item count to get to: 5000 curr count: 635
2021/12/13 20:29:53 Waiting for dst bucket item count to get to: 5000 curr count: 3135
2021/12/13 20:29:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:29:58 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 20:29:58 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 20:29:58 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:29:58 Undeploying app: bucket_op_on_update
2021/12/13 20:29:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:30:03 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:30:08 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:30:13 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:30:18 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:30:18 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:30:18 No eventing-con process running
2021/12/13 20:30:20 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:30:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:31:02 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:31:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:31:03 Starting bucket ops to source bucket
2021/12/13 20:31:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:31:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:31:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:31:26 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 20:31:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:31:31 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/13 20:31:31 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/13 20:31:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:31:31 Undeploying app: bucket_op_on_update
2021/12/13 20:31:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:31:36 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:31:41 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:31:46 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:31:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/13 20:31:51 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/13 20:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:31:51 No eventing-con process running
2021/12/13 20:31:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:32:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:32:35 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (510.94s)
=== RUN   TestOnUpdateN1QLOp
2021/12/13 20:32:45 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 20:32:45 App: TestOnUpdateN1QLOp got un-deployed
2021/12/13 20:32:45 No eventing-con process running
2021/12/13 20:32:45 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/13 20:32:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:32:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:32:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:32:59 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:33:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/13 20:33:00 Starting bucket ops to source bucket
2021/12/13 20:33:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:33:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:33:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:33:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:33:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:33:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:33:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:33:28 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 20:33:33 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 20:33:38 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 20:33:43 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/13 20:33:43 App: TestOnUpdateN1QLOp got un-deployed
2021/12/13 20:33:43 No eventing-con process running
2021/12/13 20:33:43 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/13 20:33:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:33:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:33:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:33:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:34:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:34:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (91.60s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/13 20:34:17 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 20:34:17 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/13 20:34:17 No eventing-con process running
2021/12/13 20:34:17 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/13 20:34:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:34:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:34:26 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:34:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:34:31 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:34:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:34:32 Starting bucket ops to source bucket
2021/12/13 20:34:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:34:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:34:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:34:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:34:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:34:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:34:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:35:00 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 20:35:05 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 20:35:10 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 20:35:15 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/13 20:35:15 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/13 20:35:15 No eventing-con process running
2021/12/13 20:35:15 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/13 20:35:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:35:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:35:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:35:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:35:39 Waiting for dst bucket item count to get to: 0 curr count: 4504
2021/12/13 20:35:44 Waiting for dst bucket item count to get to: 0 curr count: 2004
2021/12/13 20:35:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.54s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/13 20:35:54 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 20:35:54 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 20:35:54 No eventing-con process running
2021/12/13 20:35:54 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/13 20:35:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:36:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:36:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:36:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:36:07 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:36:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 20:36:08 Starting bucket ops to source bucket
2021/12/13 20:36:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:36:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:36:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:36:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:36:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 20:36:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:36:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:36:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:36:42 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 20:36:47 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 20:36:52 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 20:36:57 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 20:36:57 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 20:36:57 No eventing-con process running
2021/12/13 20:36:57 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/13 20:36:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:37:09 Waiting for dst bucket item count to get to: 0 curr count: 4396
2021/12/13 20:37:14 Waiting for dst bucket item count to get to: 0 curr count: 1896
2021/12/13 20:37:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:37:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:37:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.84s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/13 20:37:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 20:37:35 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/13 20:37:35 No eventing-con process running
2021/12/13 20:37:35 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/13 20:37:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:37:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:37:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:37:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:37:49 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:37:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/13 20:37:50 Starting bucket ops to source bucket
2021/12/13 20:37:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:38:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:38:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:38:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:38:13 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:38:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:38:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:38:18 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 20:38:23 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 20:38:28 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 20:38:33 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/13 20:38:33 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/13 20:38:33 No eventing-con process running
2021/12/13 20:38:33 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/13 20:38:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:38:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:38:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:38:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:38:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:38:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (91.61s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/13 20:39:07 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 20:39:07 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/13 20:39:07 No eventing-con process running
2021/12/13 20:39:07 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/13 20:39:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:39:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:39:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:39:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:39:21 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:39:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/13 20:39:22 Starting bucket ops to source bucket
2021/12/13 20:39:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:39:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:39:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:39:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:39:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:39:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:39:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:39:50 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 20:39:56 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 20:40:01 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 20:40:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/13 20:40:06 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/13 20:40:06 No eventing-con process running
2021/12/13 20:40:06 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/13 20:40:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:40:17 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 20:40:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:40:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:40:29 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:40:34 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 20:40:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (97.20s)
=== RUN   TestOnDeleteBucketOp
2021/12/13 20:40:44 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 20:40:44 App: TestOnDeleteBucketOp got un-deployed
2021/12/13 20:40:44 No eventing-con process running
2021/12/13 20:40:44 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/13 20:40:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:40:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:40:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:40:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:40:58 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:40:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 20:40:59 Starting bucket ops to source bucket
2021/12/13 20:41:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:41:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:41:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:41:25 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/13 20:41:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:41:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:41:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:41:30 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 20:41:35 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 20:41:40 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 20:41:45 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/13 20:41:45 App: TestOnDeleteBucketOp got un-deployed
2021/12/13 20:41:45 No eventing-con process running
2021/12/13 20:41:45 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/13 20:41:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:41:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:41:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:42:04 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:42:09 Waiting for dst bucket item count to get to: 0 curr count: 2496
2021/12/13 20:42:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (94.92s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/13 20:42:19 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 20:42:19 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/13 20:42:19 No eventing-con process running
2021/12/13 20:42:19 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/13 20:42:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:42:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:42:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:42:33 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:42:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/13 20:42:34 Starting bucket ops to source bucket
2021/12/13 20:42:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:42:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:42:55 Waiting for dst bucket item count to get to: 5000 curr count: 1794
2021/12/13 20:43:00 Waiting for dst bucket item count to get to: 5000 curr count: 4290
2021/12/13 20:43:00 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:43:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:43:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:43:05 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 20:43:10 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 20:43:15 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 20:43:20 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/13 20:43:20 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/13 20:43:20 No eventing-con process running
2021/12/13 20:43:20 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/13 20:43:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:43:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:43:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:43:39 Waiting for dst bucket item count to get to: 0 curr count: 4554
2021/12/13 20:43:44 Waiting for dst bucket item count to get to: 0 curr count: 2058
2021/12/13 20:43:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (94.87s)
=== RUN   TestTimerBucketOp
2021/12/13 20:43:54 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 20:43:54 App: TestTimerBucketOp got un-deployed
2021/12/13 20:43:54 No eventing-con process running
2021/12/13 20:43:54 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/13 20:43:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:44:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:44:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:44:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:44:08 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:44:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 20:44:09 Starting bucket ops to source bucket
2021/12/13 20:44:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:44:47 Waiting for dst bucket item count to get to: 5000 curr count: 2076
2021/12/13 20:44:52 Waiting for dst bucket item count to get to: 5000 curr count: 4576
2021/12/13 20:44:52 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:44:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:44:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:44:57 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 20:45:02 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 20:45:07 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 20:45:12 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/13 20:45:12 App: TestTimerBucketOp got un-deployed
2021/12/13 20:45:12 No eventing-con process running
2021/12/13 20:45:12 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/13 20:45:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:45:24 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:45:29 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 20:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:45:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:45:41 Waiting for dst bucket item count to get to: 0 curr count: 4166
2021/12/13 20:45:46 Waiting for dst bucket item count to get to: 0 curr count: 4166
2021/12/13 20:45:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (121.47s)
=== RUN   TestTimerInPastBucketOp
2021/12/13 20:45:56 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 20:45:56 App: TestTimerInPastBucketOp got un-deployed
2021/12/13 20:45:56 No eventing-con process running
2021/12/13 20:45:56 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/13 20:45:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:46:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:46:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:46:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:46:09 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:46:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 20:46:10 Starting bucket ops to source bucket
2021/12/13 20:46:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:46:48 Waiting for dst bucket item count to get to: 5000 curr count: 3883
2021/12/13 20:46:53 Waiting for dst bucket item count to get to: 5000 curr count: 3883
2021/12/13 20:46:53 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:46:53 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:46:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:46:58 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 20:47:03 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 20:47:08 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 20:47:13 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/13 20:47:13 App: TestTimerInPastBucketOp got un-deployed
2021/12/13 20:47:13 No eventing-con process running
2021/12/13 20:47:13 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/13 20:47:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:47:25 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:47:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:47:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:47:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 20:47:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (111.47s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/13 20:47:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:47:47 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:47:47 No eventing-con process running
2021/12/13 20:47:47 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/13 20:47:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:47:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:47:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:48:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:48:01 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:48:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 20:48:02 Starting bucket ops to source bucket
2021/12/13 20:48:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:48:40 Waiting for dst bucket item count to get to: 5000 curr count: 3848
2021/12/13 20:48:45 Waiting for dst bucket item count to get to: 5000 curr count: 3848
2021/12/13 20:48:45 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:48:50 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 20:48:50 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 20:48:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:48:50 Undeploying app: bucket_op_with_timer
2021/12/13 20:48:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:48:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:49:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:49:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:49:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:49:10 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:49:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:49:10 No eventing-con process running
2021/12/13 20:49:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:49:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:49:54 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:49:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 20:49:55 Starting bucket ops to source bucket
2021/12/13 20:50:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:50:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:50:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:50:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:50:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:50:28 Waiting for dst bucket item count to get to: 5000 curr count: 63
2021/12/13 20:50:33 Waiting for dst bucket item count to get to: 5000 curr count: 63
2021/12/13 20:50:33 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:50:38 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 20:50:38 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 20:50:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:50:38 Undeploying app: bucket_op_with_timer
2021/12/13 20:50:38 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:50:43 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:50:48 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:50:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:50:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:50:58 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:50:58 No eventing-con process running
2021/12/13 20:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:51:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:51:42 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:51:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 20:51:44 Starting bucket ops to source bucket
2021/12/13 20:51:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:51:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:52:27 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/13 20:52:27 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:52:32 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 20:52:32 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 20:52:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:52:32 Undeploying app: bucket_op_with_timer
2021/12/13 20:52:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:52:37 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:52:42 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:52:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:52:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:52:52 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:52:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:52:52 No eventing-con process running
2021/12/13 20:52:54 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:53:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:53:36 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:53:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 20:53:37 Starting bucket ops to source bucket
2021/12/13 20:53:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:53:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:53:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:54:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:54:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:54:10 Waiting for dst bucket item count to get to: 5000 curr count: 395
2021/12/13 20:54:15 Waiting for dst bucket item count to get to: 5000 curr count: 395
2021/12/13 20:54:20 Waiting for dst bucket item count to get to: 5000 curr count: 3938
2021/12/13 20:54:25 Waiting for dst bucket item count to get to: 5000 curr count: 3938
2021/12/13 20:54:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:54:30 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 20:54:30 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 20:54:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:54:30 Undeploying app: bucket_op_with_timer
2021/12/13 20:54:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:54:35 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:54:40 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:54:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:54:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:54:50 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:54:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:54:50 No eventing-con process running
2021/12/13 20:54:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:54:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:55:34 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:55:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 20:55:35 Starting bucket ops to source bucket
2021/12/13 20:55:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:55:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:55:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:55:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:56:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:56:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:56:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:56:18 Waiting for dst bucket item count to get to: 5000 curr count: 4950
2021/12/13 20:56:23 Waiting for dst bucket item count to get to: 5000 curr count: 4950
2021/12/13 20:56:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:56:28 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/13 20:56:28 App: TestDeployUndeployLoopTimer got deployed
2021/12/13 20:56:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:56:28 Undeploying app: bucket_op_with_timer
2021/12/13 20:56:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:56:33 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:56:38 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:56:44 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:56:49 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/13 20:56:49 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/13 20:56:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:56:49 No eventing-con process running
2021/12/13 20:56:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:56:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:57:33 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (595.67s)
=== RUN   TestMultipleHandlers
2021/12/13 20:57:43 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 20:57:43 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/13 20:57:43 No eventing-con process running
2021/12/13 20:57:43 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/13 20:57:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:57:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:57:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:58:01 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 20:58:01 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/13 20:58:01 No eventing-con process running
2021/12/13 20:58:01 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/13 20:58:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:58:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:58:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 20:58:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 20:58:15 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:58:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 20:58:16 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 20:58:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/13 20:58:22 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 20:58:27 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 20:58:32 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 20:58:37 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/13 20:58:37 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/13 20:58:42 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 20:58:47 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 20:58:52 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/13 20:58:52 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/13 20:58:52 Starting bucket ops to source bucket
2021/12/13 20:59:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 20:59:07 Waiting for dst bucket item count to get to: 5000 curr count: 4979
2021/12/13 20:59:12 Waiting for dst bucket item count to get to: 5000 curr count: 4979
2021/12/13 20:59:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 20:59:12 Starting bucket ops to source bucket
2021/12/13 20:59:27 Waiting for dst bucket item count to get to: 10000 curr count: 6074
2021/12/13 20:59:32 Waiting for dst bucket item count to get to: 10000 curr count: 6074
2021/12/13 20:59:32 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 20:59:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 20:59:32 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 20:59:33 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 20:59:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 20:59:38 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 20:59:43 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 20:59:48 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 20:59:53 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/13 20:59:53 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/13 20:59:53 No eventing-con process running
2021/12/13 20:59:53 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/13 20:59:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:00:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:00:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:00:12 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:00:17 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:00:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:00:22 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 21:00:27 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 21:00:32 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 21:00:37 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/13 21:00:37 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/13 21:00:37 No eventing-con process running
2021/12/13 21:00:37 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/13 21:00:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:00:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.09s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/13 21:01:01 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 21:01:01 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/13 21:01:01 No eventing-con process running
2021/12/13 21:01:01 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/13 21:01:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:01:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:01:09 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:01:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:01:14 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:01:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 21:01:15 Starting bucket ops to source bucket
2021/12/13 21:01:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:01:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:01:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:01:38 Waiting for dst bucket item count to get to: 5000 curr count: 3007
2021/12/13 21:01:43 Waiting for dst bucket item count to get to: 5000 curr count: 3007
2021/12/13 21:01:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:01:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:01:49 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 21:01:49 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 21:01:49 Pausing app: bucket_op_on_update
2021/12/13 21:01:49 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:01:54 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 21:01:54 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 21:01:54 Resuming app: bucket_op_on_update
2021/12/13 21:01:55 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:01:55 Starting bucket ops to source bucket
2021/12/13 21:02:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:02:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:02:13 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:02:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 21:02:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:02:18 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 21:02:18 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 21:02:18 Pausing app: bucket_op_on_update
2021/12/13 21:02:18 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:02:23 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 21:02:23 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 21:02:23 Resuming app: bucket_op_on_update
2021/12/13 21:02:24 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:02:24 Starting bucket ops to source bucket
2021/12/13 21:02:32 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:02:37 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:02:42 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:02:42 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/13 21:02:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:02:47 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 21:02:47 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 21:02:47 Pausing app: bucket_op_on_update
2021/12/13 21:02:47 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:02:52 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 21:02:52 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 21:02:52 Resuming app: bucket_op_on_update
2021/12/13 21:02:53 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:02:53 Starting bucket ops to source bucket
2021/12/13 21:03:01 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:03:06 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:03:11 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:03:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 21:03:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:03:16 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 21:03:16 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 21:03:16 Pausing app: bucket_op_on_update
2021/12/13 21:03:17 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:03:22 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 21:03:22 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 21:03:22 Resuming app: bucket_op_on_update
2021/12/13 21:03:23 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:03:23 Starting bucket ops to source bucket
2021/12/13 21:03:31 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:03:36 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:03:41 Waiting for dst bucket item count to get to: 25000 curr count: 21173
2021/12/13 21:03:46 Waiting for dst bucket item count to get to: 25000 curr count: 21173
2021/12/13 21:03:46 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 21:03:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:03:51 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/13 21:03:51 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/13 21:03:51 Pausing app: bucket_op_on_update
2021/12/13 21:03:51 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:03:56 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/13 21:03:56 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/13 21:03:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:04:01 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 21:04:06 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 21:04:11 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 21:04:16 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/13 21:04:16 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/13 21:04:16 No eventing-con process running
2021/12/13 21:04:16 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/13 21:04:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:04:28 Waiting for dst bucket item count to get to: 0 curr count: 15267
2021/12/13 21:04:33 Waiting for dst bucket item count to get to: 0 curr count: 2767
2021/12/13 21:04:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:04:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:04:45 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 21:04:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (233.97s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/13 21:04:55 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 21:04:55 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/13 21:04:55 No eventing-con process running
2021/12/13 21:04:55 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/13 21:04:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:05:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:05:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:05:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:05:08 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:05:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 21:05:09 Starting bucket ops to source bucket
2021/12/13 21:05:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:05:47 Waiting for dst bucket item count to get to: 5000 curr count: 1803
2021/12/13 21:05:52 Waiting for dst bucket item count to get to: 5000 curr count: 4283
2021/12/13 21:05:57 Waiting for dst bucket item count to get to: 5000 curr count: 4980
2021/12/13 21:05:57 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:05:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:06:02 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 21:06:02 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 21:06:02 Pausing app: bucket_op_with_timer
2021/12/13 21:06:03 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:06:08 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 21:06:08 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 21:06:08 Resuming app: bucket_op_with_timer
2021/12/13 21:06:09 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:06:09 Starting bucket ops to source bucket
2021/12/13 21:06:17 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:22 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:27 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:32 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:37 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:42 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:06:42 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 21:06:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:06:47 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 21:06:47 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 21:06:47 Pausing app: bucket_op_with_timer
2021/12/13 21:06:47 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:06:52 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 21:06:52 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 21:06:52 Resuming app: bucket_op_with_timer
2021/12/13 21:06:53 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:06:53 Starting bucket ops to source bucket
2021/12/13 21:07:01 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:07:06 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:07:11 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/13 21:07:16 Waiting for dst bucket item count to get to: 15000 curr count: 10786
2021/12/13 21:07:21 Waiting for dst bucket item count to get to: 15000 curr count: 10786
2021/12/13 21:07:26 Waiting for dst bucket item count to get to: 15000 curr count: 14095
2021/12/13 21:07:31 Waiting for dst bucket item count to get to: 15000 curr count: 14095
2021/12/13 21:07:31 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/13 21:07:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:07:36 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 21:07:36 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 21:07:36 Pausing app: bucket_op_with_timer
2021/12/13 21:07:37 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:07:42 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 21:07:42 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 21:07:42 Resuming app: bucket_op_with_timer
2021/12/13 21:07:43 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:07:43 Starting bucket ops to source bucket
2021/12/13 21:07:51 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:07:56 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:08:01 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:08:06 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/13 21:08:11 Waiting for dst bucket item count to get to: 20000 curr count: 16437
2021/12/13 21:08:16 Waiting for dst bucket item count to get to: 20000 curr count: 16437
2021/12/13 21:08:16 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 21:08:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:08:21 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 21:08:21 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 21:08:21 Pausing app: bucket_op_with_timer
2021/12/13 21:08:21 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:08:26 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 21:08:26 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 21:08:26 Resuming app: bucket_op_with_timer
2021/12/13 21:08:27 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:08:27 Starting bucket ops to source bucket
2021/12/13 21:08:35 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:08:40 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:08:45 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:08:50 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:08:55 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/13 21:09:00 Waiting for dst bucket item count to get to: 25000 curr count: 21719
2021/12/13 21:09:05 Waiting for dst bucket item count to get to: 25000 curr count: 21719
2021/12/13 21:09:05 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 21:09:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:09:10 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/13 21:09:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/13 21:09:10 Pausing app: bucket_op_with_timer
2021/12/13 21:09:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:09:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/13 21:09:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/13 21:09:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:09:20 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 21:09:25 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 21:09:30 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 21:09:35 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/13 21:09:35 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/13 21:09:35 No eventing-con process running
2021/12/13 21:09:35 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/13 21:09:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:09:47 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 21:09:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:09:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:09:59 Waiting for dst bucket item count to get to: 0 curr count: 23810
2021/12/13 21:10:04 Waiting for dst bucket item count to get to: 0 curr count: 11310
2021/12/13 21:10:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.11s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/13 21:10:14 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 21:10:14 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/13 21:10:14 No eventing-con process running
2021/12/13 21:10:14 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/13 21:10:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:10:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:10:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:10:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:10:27 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:10:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/13 21:10:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 21:10:39 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 21:10:44 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 21:10:49 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/13 21:10:49 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/13 21:10:49 Starting bucket ops to source bucket
2021/12/13 21:10:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:11:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:11:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:11:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:11:18 Waiting for dst bucket item count to get to: 5000 curr count: 2306
2021/12/13 21:11:23 Waiting for dst bucket item count to get to: 5000 curr count: 4806
2021/12/13 21:11:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:11:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:11:23 Pausing app: bucket_op_with_timer
2021/12/13 21:11:23 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:11:28 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/13 21:11:28 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/13 21:11:28 Resuming app: bucket_op_with_timer
2021/12/13 21:11:29 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:11:29 Starting bucket ops to source bucket
2021/12/13 21:11:41 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:11:46 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:11:51 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:11:56 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:12:01 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:12:06 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/13 21:12:06 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 21:12:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:12:11 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 21:12:16 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 21:12:21 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 21:12:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/13 21:12:26 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/13 21:12:26 No eventing-con process running
2021/12/13 21:12:26 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/13 21:12:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:12:38 Waiting for dst bucket item count to get to: 0 curr count: 7593
2021/12/13 21:12:43 Waiting for dst bucket item count to get to: 0 curr count: 2593
2021/12/13 21:12:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:12:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:12:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (171.18s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/13 21:13:05 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 21:13:05 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/13 21:13:05 No eventing-con process running
2021/12/13 21:13:05 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/13 21:13:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:13:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:13:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:13:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:13:19 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:13:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 21:13:25 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 21:13:30 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 21:13:35 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 21:13:40 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/13 21:13:40 App: TestPauseResumeWithEventingReb got deployed
2021/12/13 21:13:40 Starting bucket ops to source bucket
2021/12/13 21:13:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:13:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:13:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:14:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:14:09 Waiting for dst bucket item count to get to: 5000 curr count: 396
2021/12/13 21:14:14 Waiting for dst bucket item count to get to: 5000 curr count: 791
2021/12/13 21:14:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:14:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:14:14 Pausing app: bucket_op_with_timer
2021/12/13 21:14:14 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:14:19 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/13 21:14:19 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/13 21:14:19 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 21:14:20 Removing node(s): [] from the cluster
2021/12/13 21:14:25 Rebalance progress: 29.16666666666666
2021/12/13 21:14:30 Rebalance progress: 29.16666666666666
2021/12/13 21:14:35 Rebalance progress: 100
2021/12/13 21:14:35 Resuming app: bucket_op_with_timer
2021/12/13 21:14:36 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:14:36 Starting bucket ops to source bucket
2021/12/13 21:14:48 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:14:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:14:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:15:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/13 21:15:08 Waiting for dst bucket item count to get to: 10000 curr count: 6648
2021/12/13 21:15:13 Waiting for dst bucket item count to get to: 10000 curr count: 6648
2021/12/13 21:15:13 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 21:15:13 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 21:15:18 Rebalance progress: 29.16666666666666
2021/12/13 21:15:23 Rebalance progress: 29.16666666666666
2021/12/13 21:15:28 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/13 21:15:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:15:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:16:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:16:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:16:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:16:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:17:03 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:17:18 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:17:33 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 21:17:48 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:81} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 380e3557fb5e287436052c7c7a803448
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
Producer node: 172.31.2.134:9300	Node UUID: e819be24711971bbc2204265433dadcd
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
Producer node: 127.0.0.1:9303	Node UUID: af7660ed7963dc5cc343f4149d950c83
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]


2021/12/13 21:18:03 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:18:03 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:18:08 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 21:18:13 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 21:18:18 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 21:18:23 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/13 21:18:23 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/13 21:18:23 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:18:23 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/13 21:18:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:18:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:18:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:18:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:18:47 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:18:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (351.95s)
=== RUN   TestBucketDeleteAfterPause
2021/12/13 21:18:57 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/13 21:18:57 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/13 21:18:57 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:18:57 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/13 21:18:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:19:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:19:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:19:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:19:11 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:19:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/13 21:19:17 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 21:19:22 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 21:19:27 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 21:19:32 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/13 21:19:32 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/13 21:19:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:19:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:19:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:19:41 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:19:41 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/13 21:19:42 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:19:47 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/13 21:19:47 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/13 21:19:47 handler_test.go:716: Deleting source bucket: default
2021/12/13 21:19:53 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/13 21:19:53 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/13 21:19:58 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/13 21:19:58 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/13 21:19:58 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:19:58 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/13 21:20:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:20:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:20:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:20:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:20:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (89.41s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/13 21:20:26 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 21:20:26 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/13 21:20:26 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:20:26 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/13 21:20:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:20:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:20:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:20:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:20:40 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:20:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/13 21:20:46 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 21:20:51 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 21:20:56 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 21:21:01 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/13 21:21:01 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/13 21:21:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:21:36 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/13 21:21:36 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:21:41 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/13 21:21:41 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/13 21:24:41 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/13 21:24:41 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/13 21:24:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/13 21:24:47 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 21:24:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:24:47 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/13 21:24:52 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 21:24:52 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:24:52 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 2965 expected 5000
2021/12/13 21:24:57 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 21:24:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:24:57 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3901 expected 5000
2021/12/13 21:25:02 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/13 21:25:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:25:02 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/13 21:25:02 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/13 21:25:03 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:25:13 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 21:25:18 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 21:25:23 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/13 21:25:23 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/13 21:25:23 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:25:23 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/13 21:25:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:25:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:25:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:25:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:25:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.37s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/13 21:25:56 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 21:25:56 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/13 21:25:56 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:25:56 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/13 21:25:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:26:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:26:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:26:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:26:09 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:26:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 21:26:15 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:26:20 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:26:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:26:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:26:30 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/13 21:26:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:26:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:26:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:26:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:26:44 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:26:44 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/13 21:26:44 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:27:14 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/13 21:27:15 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 21:27:20 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:27:25 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:27:30 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:27:35 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/13 21:27:35 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/13 21:27:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:27:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4989
2021/12/13 21:27:53 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4989
2021/12/13 21:27:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:27:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:27:53 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/13 21:27:53 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:28:03 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 21:28:08 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 21:28:13 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/13 21:28:13 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/13 21:28:13 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:28:13 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/13 21:28:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:28:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:28:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:28:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (160.43s)
=== RUN   TestCPPWorkerCleanup
2021/12/13 21:28:36 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 21:28:36 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/13 21:28:36 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:28:36 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/13 21:28:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:28:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:28:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:28:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:28:49 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:28:50 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/13 21:28:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:28:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:29:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:29:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:29:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:29:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4814
2021/12/13 21:29:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4814
2021/12/13 21:29:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:29:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:29:23 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:29:28 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 21:29:33 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 21:29:38 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 21:29:43 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/13 21:29:43 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/13 21:29:43 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:29:43 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/13 21:29:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:29:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:29:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:29:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:30:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:30:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4703
2021/12/13 21:30:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4703
2021/12/13 21:30:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (139.72s)
=== RUN   TestWithUserXattrs
2021/12/13 21:30:56 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 21:30:56 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/13 21:30:56 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:30:56 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/13 21:30:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:31:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:31:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:31:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:31:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:31:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:31:13 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:31:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/13 21:31:19 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 21:31:24 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 21:31:29 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 21:31:34 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/13 21:31:34 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/13 21:31:34 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:31:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:31:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4134
2021/12/13 21:31:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4134
2021/12/13 21:31:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:31:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:31:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:31:59 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 21:32:04 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 21:32:09 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 21:32:14 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/13 21:32:14 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/13 21:32:14 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:32:14 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/13 21:32:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:32:25 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:32:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:32:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:32:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:32:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.04s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/13 21:32:47 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:32:47 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 21:32:47 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:32:47 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/13 21:32:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:32:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:32:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:33:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:33:00 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:33:01 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/13 21:33:06 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 21:33:11 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 21:33:16 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 21:33:21 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/13 21:33:21 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/13 21:33:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:33:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:33:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:33:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:33:35 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:33:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:33:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3084
2021/12/13 21:33:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 584
2021/12/13 21:33:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:33:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:34:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:34:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:34:05 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:34:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15212
2021/12/13 21:34:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 15212
2021/12/13 21:34:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/13 21:34:36 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:34:36 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:34:41 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:34:46 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:34:51 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:34:56 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:34:56 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 21:34:56 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:35:01 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/13 21:35:01 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/13 21:35:01 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:35:01 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/13 21:35:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:35:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/13 21:35:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/13 21:35:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:35:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:35:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/13 21:35:34 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/13 21:35:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (177.42s)
=== RUN   TestMetaBucketDelete
2021/12/13 21:35:44 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 21:35:44 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 21:35:44 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:35:44 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/13 21:35:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:35:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:35:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:35:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:35:57 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:35:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 21:36:04 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 21:36:09 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 21:36:14 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 21:36:19 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/13 21:36:19 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/13 21:36:19 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/13 21:36:20 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/13 21:36:25 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 21:36:25 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 21:36:40 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/13 21:36:40 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/13 21:36:40 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:36:40 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/13 21:36:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:36:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:36:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:36:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.87s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/13 21:37:03 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:37:03 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:37:03 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:37:03 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 21:37:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:37:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:37:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:37:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:37:16 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:37:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 21:37:36 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/13 21:37:37 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/13 21:37:52 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:37:52 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:38:07 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:38:07 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:38:07 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:38:07 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 21:38:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:38:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:38:15 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:38:20 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (87.07s)
=== RUN   TestSourceBucketDelete
2021/12/13 21:38:30 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 21:38:30 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 21:38:30 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:38:30 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/13 21:38:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:38:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:38:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:38:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:38:43 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:38:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/13 21:38:50 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 21:38:55 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 21:39:00 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 21:39:05 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/13 21:39:05 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/13 21:39:05 handler_test.go:945: Deleting source bucket: default
2021/12/13 21:39:06 handler_test.go:947: Deleted source bucket: default
2021/12/13 21:39:11 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 21:39:11 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 21:39:26 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/13 21:39:26 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/13 21:39:26 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:39:26 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/13 21:39:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:39:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:39:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:39:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (78.84s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/13 21:39:49 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:39:49 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:39:49 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:39:49 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 21:39:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:39:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:39:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:40:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:40:02 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:40:03 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 21:40:22 handler_test.go:966: Deleting source bucket: default
2021/12/13 21:40:24 handler_test.go:968: Deleted source bucket: default
2021/12/13 21:40:39 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:40:39 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:40:54 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:40:54 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:40:54 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:40:54 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 21:40:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:41:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:41:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:41:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.69s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/13 21:41:17 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:41:17 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:41:17 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:41:17 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/13 21:41:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:41:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:41:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:41:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:41:30 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:41:31 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 21:41:50 handler_test.go:988: Deleting source bucket: default
2021/12/13 21:41:51 handler_test.go:990: Deleted source bucket: default
2021/12/13 21:41:51 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/13 21:41:52 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/13 21:42:07 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:42:07 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:42:22 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/13 21:42:22 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/13 21:42:22 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:42:22 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/13 21:42:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:42:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:42:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:42:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.22s)
=== RUN   TestUndeployDuringBootstrap
2021/12/13 21:42:45 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 21:42:45 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 21:42:45 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:42:45 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/13 21:42:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:42:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:42:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:42:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:42:58 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:42:59 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 21:42:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:43:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:43:11 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:43:11 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 21:43:11 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/13 21:43:11 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 21:43:16 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/13 21:43:21 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/13 21:43:26 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 21:43:31 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 21:43:36 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 21:43:36 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 21:43:37 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:44:02 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/13 21:44:02 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/13 21:44:02 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:44:02 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/13 21:44:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:44:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:44:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:44:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:44:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:44:30 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1367
2021/12/13 21:44:35 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1367
2021/12/13 21:44:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (119.57s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/13 21:44:40 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:44:41 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/13 21:44:46 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 21:44:51 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 21:44:56 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 21:45:01 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/13 21:45:01 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/13 21:45:01 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/13 21:45:01 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:45:01 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/13 21:45:06 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/13 21:45:06 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/13 21:45:06 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:45:06 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/13 21:45:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:45:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:45:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:45:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.58s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/13 21:45:29 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:45:29 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 21:45:29 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:45:29 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/13 21:45:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:45:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:45:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:45:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:45:42 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:45:44 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/13 21:45:49 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 21:45:54 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 21:45:59 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 21:46:04 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/13 21:46:04 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/13 21:46:04 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:46:34 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:46:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:46:44 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:46:49 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:46:54 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:46:54 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 21:46:54 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:48:39 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/13 21:48:39 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/13 21:48:39 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:48:39 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/13 21:48:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:48:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4775
2021/12/13 21:48:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4775
2021/12/13 21:48:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:48:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:49:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:49:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.18s)
=== RUN   TestUndeployWithKVFailover
2021/12/13 21:49:17 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 21:49:17 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/13 21:49:17 eventingSetup.go:46: No eventing-con process running
2021/12/13 21:49:17 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/13 21:49:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:49:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:49:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:49:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:49:31 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:49:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/13 21:49:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 21:49:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:49:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:49:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:49:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:49:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:50:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:50:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1907
2021/12/13 21:50:14 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1907
2021/12/13 21:50:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:50:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 21:50:14 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 21:50:19 Rebalance progress: 29.16666666666666
2021/12/13 21:50:24 Rebalance progress: 29.16666666666666
2021/12/13 21:50:29 Rebalance progress: 100
2021/12/13 21:50:29 Failing over: [127.0.0.1:19003]
2021/12/13 21:50:39 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/13 21:51:39 Removing node(s): [] from the cluster
2021/12/13 21:51:44 Rebalance progress: 29.16666666666666
2021/12/13 21:51:49 Rebalance progress: 29.16666666666666
2021/12/13 21:51:54 Rebalance progress: 100
2021/12/13 21:51:59 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 21:51:59 App: TestUndeployWithKVFailover got un-deployed
2021/12/13 21:51:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:52:04 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/13 21:52:04 App: TestUndeployWithKVFailover got un-deployed
2021/12/13 21:52:04 No eventing-con process running
2021/12/13 21:52:04 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/13 21:52:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:52:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:52:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:52:27 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (200.13s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/13 21:52:38 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 21:52:38 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/13 21:52:38 No eventing-con process running
2021/12/13 21:52:38 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/13 21:52:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:52:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:52:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:52:51 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:52:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/13 21:52:57 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 21:53:02 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 21:53:07 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 21:53:12 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/13 21:53:12 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/13 21:53:12 Starting bucket ops to source bucket
2021/12/13 21:53:32 Waiting for dst bucket item count to get to: 20000 curr count: 8268
2021/12/13 21:53:32 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/13 21:53:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:53:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:53:45 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/13 21:53:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:53:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:53:56 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/13 21:53:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:53:56 Starting bucket ops to source bucket
2021/12/13 21:54:10 Waiting for dst bucket item count to get to: 10000 curr count: 2098
2021/12/13 21:54:15 Waiting for dst bucket item count to get to: 10000 curr count: 2098
2021/12/13 21:54:15 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 21:54:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:54:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:54:20 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 21:54:25 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 21:54:30 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 21:54:35 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/13 21:54:35 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/13 21:54:35 No eventing-con process running
2021/12/13 21:54:35 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/13 21:54:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:54:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 21:54:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:54:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:54:58 Waiting for dst bucket item count to get to: 0 curr count: 7022
2021/12/13 21:55:03 Waiting for dst bucket item count to get to: 0 curr count: 2022
2021/12/13 21:55:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (155.55s)
=== RUN   TestTimerOverwrite
2021/12/13 21:55:13 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 21:55:13 App: TestTimerOverwrite got un-deployed
2021/12/13 21:55:13 No eventing-con process running
2021/12/13 21:55:13 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/13 21:55:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:55:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:55:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:55:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:55:26 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:55:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/13 21:55:32 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 21:55:37 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 21:55:42 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 21:55:47 Waiting for app: TestTimerOverwrite to get deployed
2021/12/13 21:55:47 App: TestTimerOverwrite got deployed
2021/12/13 21:55:57 Starting bucket ops to source bucket
2021/12/13 21:56:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:37 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:42 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 21:56:57 Waiting for dst bucket item count to get to: 5000 curr count: 539
2021/12/13 21:57:02 Waiting for dst bucket item count to get to: 5000 curr count: 1074
2021/12/13 21:57:07 Waiting for dst bucket item count to get to: 5000 curr count: 3000
2021/12/13 21:57:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 21:57:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:57:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:57:22 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 21:57:27 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 21:57:32 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 21:57:37 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/13 21:57:37 App: TestTimerOverwrite got un-deployed
2021/12/13 21:57:37 No eventing-con process running
2021/12/13 21:57:37 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/13 21:57:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:57:48 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:57:53 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 21:57:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:57:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:58:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 21:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (182.23s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/13 21:58:10 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/13 21:58:10 Removing node(s): [] from the cluster
2021/12/13 21:58:16 Rebalance progress: 100
2021/12/13 21:58:16 Failing over: [127.0.0.1:19003]
2021/12/13 21:58:21 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 21:58:21 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 21:58:21 No eventing-con process running
2021/12/13 21:58:21 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/13 21:58:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:58:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:58:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 21:58:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 21:58:34 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/13 21:58:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/13 21:58:35 Starting bucket ops to source bucket
2021/12/13 21:58:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:58:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 21:58:45 Context: Post to set retry counter, response code: 200 dump: 
2021/12/13 21:58:45 set retry counter, response &{[] }
2021/12/13 21:59:50 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 21:59:50 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 21:59:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 21:59:50 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/13 21:59:55 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/13 21:59:55 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/13 21:59:55 No eventing-con process running
2021/12/13 21:59:55 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/13 21:59:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:00:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 22:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:00:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:00:18 Waiting for dst bucket item count to get to: 0 curr count: 1947
2021/12/13 22:00:23 Waiting for dst bucket item count to get to: 0 curr count: 281
2021/12/13 22:00:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:00:23 Removing node(s): [] from the cluster
2021/12/13 22:00:28 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (142.85s)
=== RUN   TestOnUpdateSrcMutation
2021/12/13 22:00:38 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 22:00:38 App: TestOnUpdateSrcMutation got un-deployed
2021/12/13 22:00:38 No eventing-con process running
2021/12/13 22:00:38 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/13 22:00:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:00:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:00:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:00:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:00:51 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:00:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/13 22:00:52 Starting bucket ops to source bucket
2021/12/13 22:01:14 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 22:01:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:01:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:01:20 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 22:01:25 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 22:01:30 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 22:01:35 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/13 22:01:35 App: TestOnUpdateSrcMutation got un-deployed
2021/12/13 22:01:35 No eventing-con process running
2021/12/13 22:01:35 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/13 22:01:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:01:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 22:01:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:01:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:01:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (84.54s)
=== RUN   TestOnDeleteSrcMutation
2021/12/13 22:02:03 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 22:02:03 App: TestOnDeleteSrcMutation got un-deployed
2021/12/13 22:02:03 No eventing-con process running
2021/12/13 22:02:03 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/13 22:02:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:02:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:02:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:02:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:02:16 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:02:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/13 22:02:17 Starting bucket ops to source bucket
2021/12/13 22:02:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 22:02:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:02:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:02:51 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 22:02:56 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 22:03:01 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 22:03:06 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/13 22:03:06 App: TestOnDeleteSrcMutation got un-deployed
2021/12/13 22:03:06 No eventing-con process running
2021/12/13 22:03:06 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/13 22:03:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:03:17 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 22:03:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:03:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:03:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.25s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/13 22:03:34 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 22:03:34 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/13 22:03:34 No eventing-con process running
2021/12/13 22:03:34 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/13 22:03:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:03:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:03:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:03:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:03:47 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:03:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 22:03:48 Starting bucket ops to source bucket
2021/12/13 22:04:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 22:04:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:04:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:04:30 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 22:04:35 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 22:04:40 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 22:04:45 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/13 22:04:45 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/13 22:04:45 No eventing-con process running
2021/12/13 22:04:45 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/13 22:04:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:04:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 22:04:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:04:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:05:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (99.62s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/13 22:05:14 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 22:05:14 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/13 22:05:14 No eventing-con process running
2021/12/13 22:05:14 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/13 22:05:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:05:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:05:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:05:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:05:27 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:05:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/13 22:05:28 Starting bucket ops to source bucket
2021/12/13 22:06:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 22:06:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:06:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:06:12 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 22:06:17 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 22:06:22 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 22:06:27 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/13 22:06:27 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/13 22:06:27 No eventing-con process running
2021/12/13 22:06:27 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/13 22:06:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:06:38 Waiting for dst bucket item count to get to: 0 curr count: 3839
2021/12/13 22:06:43 Waiting for dst bucket item count to get to: 0 curr count: 1339
2021/12/13 22:06:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:06:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:06:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (106.35s)
=== RUN   TestInterHandlerRecursion
2021/12/13 22:07:00 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:07:00 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 22:07:00 No eventing-con process running
2021/12/13 22:07:00 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/13 22:07:01 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:07:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:07:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:07:18 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:07:18 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 22:07:18 No eventing-con process running
2021/12/13 22:07:18 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 22:07:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:07:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:07:26 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:07:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:07:31 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:07:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/13 22:07:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:07:38 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:07:43 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:07:48 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:07:53 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:07:53 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/13 22:07:53 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:07:53 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/13 22:07:53 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/13 22:07:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:07:58 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:08:03 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:08:08 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:08:13 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:08:13 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 22:08:13 No eventing-con process running
2021/12/13 22:08:13 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/13 22:08:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:08:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:08:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:08:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:08:31 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:08:31 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 22:08:31 No eventing-con process running
2021/12/13 22:08:31 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 22:08:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:08:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:08:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:08:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.03s)
=== RUN   TestInterBucketRecursion
2021/12/13 22:08:49 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:08:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:08:50 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:08:50 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:08:50 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/13 22:08:55 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 22:09:00 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 22:09:05 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 22:09:10 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/13 22:09:10 App: TestInterBucketRecursion_function1 got deployed
2021/12/13 22:09:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:09:15 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:09:20 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:09:25 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:09:30 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:09:30 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/13 22:09:30 No eventing-con process running
2021/12/13 22:09:30 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/13 22:09:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:09:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:09:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:09:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:09:48 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/13 22:09:48 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/13 22:09:48 No eventing-con process running
2021/12/13 22:09:48 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/13 22:09:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:09:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:09:56 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:10:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.48s)
=== RUN   TestLargeHandler
2021/12/13 22:10:01 Changed force_compress value to false
2021/12/13 22:10:01 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:10:02 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/13 22:10:02 Changed force_compress value to true
2021/12/13 22:10:02 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:10:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/13 22:10:08 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 22:10:13 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 22:10:18 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 22:10:23 Waiting for app: TestLargeHandler_function to get deployed
2021/12/13 22:10:23 App: TestLargeHandler_function got deployed
2021/12/13 22:10:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:10:29 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 22:10:34 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 22:10:39 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 22:10:44 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/13 22:10:44 App: TestLargeHandler_function got un-deployed
2021/12/13 22:10:44 No eventing-con process running
2021/12/13 22:10:44 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/13 22:10:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:10:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:10:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:10:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.33s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/13 22:11:07 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:11:07 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 22:11:07 No eventing-con process running
2021/12/13 22:11:07 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/13 22:11:08 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:11:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:11:15 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:11:25 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:11:25 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 22:11:25 No eventing-con process running
2021/12/13 22:11:25 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/13 22:11:26 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:11:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:11:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:11:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:11:38 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:11:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/13 22:11:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:11:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:11:49 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:11:54 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:11:59 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/13 22:11:59 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/13 22:11:59 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:12:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/13 22:12:00 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:12:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 22:12:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 22:12:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 22:12:20 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/13 22:12:20 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/13 22:12:20 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/13 22:12:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:12:26 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:12:31 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:12:36 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:12:41 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/13 22:12:41 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/13 22:12:41 No eventing-con process running
2021/12/13 22:12:41 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/13 22:12:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:12:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:12:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:12:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:12:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:13:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:13:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:13:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:13:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/13 22:13:15 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/13 22:13:15 No eventing-con process running
2021/12/13 22:13:15 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/13 22:13:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:13:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:13:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:13:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.38s)
=== RUN   TestAllowInterBucketRecursion
2021/12/13 22:13:33 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:13:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:13:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:13:39 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 22:13:44 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 22:13:49 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 22:13:54 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/13 22:13:54 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/13 22:13:54 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:13:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 22:13:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:14:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 22:14:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 22:14:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 22:14:15 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/13 22:14:15 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/13 22:14:15 TestAllowInterBucketRecursion_function2 is deployed
2021/12/13 22:14:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:14:21 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:14:26 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:14:31 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:14:36 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/13 22:14:36 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/13 22:14:36 No eventing-con process running
2021/12/13 22:14:36 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/13 22:14:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:14:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:14:45 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:14:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:14:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:14:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 22:15:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 22:15:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 22:15:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/13 22:15:10 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/13 22:15:10 No eventing-con process running
2021/12/13 22:15:10 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/13 22:15:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:15:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.12s)
=== RUN   TestInterBucketRecursion2
2021/12/13 22:15:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:15:53 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:15:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:15:59 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 22:16:04 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 22:16:09 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 22:16:14 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/13 22:16:14 App: TestInterBucketRecursion2_function1 got deployed
2021/12/13 22:16:14 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:16:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 22:16:21 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 22:16:26 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 22:16:31 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 22:16:36 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/13 22:16:36 App: TestInterBucketRecursion2_function2 got deployed
2021/12/13 22:16:36 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:16:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 22:16:42 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 22:16:47 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 22:16:52 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 22:16:57 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/13 22:16:57 App: TestInterBucketRecursion2_function3 got deployed
2021/12/13 22:16:57 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:16:57 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/13 22:16:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:17:02 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 22:17:07 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 22:17:12 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 22:17:17 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/13 22:17:17 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/13 22:17:17 No eventing-con process running
2021/12/13 22:17:17 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/13 22:17:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:17:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:17:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:17:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:17:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:17:35 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 22:17:40 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 22:17:45 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 22:17:50 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/13 22:17:50 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/13 22:17:50 No eventing-con process running
2021/12/13 22:17:50 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/13 22:17:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:17:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:17:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:18:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:18:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:18:09 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 22:18:14 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 22:18:19 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 22:18:24 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/13 22:18:24 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/13 22:18:24 No eventing-con process running
2021/12/13 22:18:24 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/13 22:18:25 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:18:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:18:32 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:18:42 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/13 22:18:42 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/13 22:18:42 No eventing-con process running
2021/12/13 22:18:42 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/13 22:18:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:18:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:18:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:18:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.92s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/13 22:19:02 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 22:19:07 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/13 22:19:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:19:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:19:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:19:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:20:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:20:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:20:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:20:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:21:10 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:21:25 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:21:40 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/13 22:21:55 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:49} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/13 22:22:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:22:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 22:22:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 22:22:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/13 22:22:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:22:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 22:22:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 22:22:22 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/13 22:22:22 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:22:27 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/13 22:22:27 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/13 22:22:27 eventingSetup.go:46: No eventing-con process running
2021/12/13 22:22:27 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/13 22:22:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:22:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:22:35 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:22:40 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:22:45 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:22:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 22:22:51 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 22:22:56 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 22:23:01 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 22:23:06 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/13 22:23:06 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: 380e3557fb5e287436052c7c7a803448
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: e819be24711971bbc2204265433dadcd
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
Producer node: 127.0.0.1:9303	Node UUID: af7660ed7963dc5cc343f4149d950c83
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/13 22:23:06 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:23:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 22:23:11 Rebalance progress: 29.16666666666666
2021/12/13 22:23:16 Rebalance progress: 29.16666666666666
2021/12/13 22:23:21 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9303	Node UUID: af7660ed7963dc5cc343f4149d950c83
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]
Producer node: 127.0.0.1:9302	Node UUID: 380e3557fb5e287436052c7c7a803448
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: e819be24711971bbc2204265433dadcd
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]


2021/12/13 22:23:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:23:21 handler_test.go:1674: Deleting source bucket: default
2021/12/13 22:23:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 22:23:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 22:23:29 Rebalance progress: 29.16666666666666
2021/12/13 22:23:34 Rebalance progress: 29.16666666666666
2021/12/13 22:23:39 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 900
	vb list dump: [0, 2-6, 8-10, 12-13, 15-26, 28-30, 32-37, 39-65, 67-74, 76-84, 86-104, 106-107, 109-112, 114-133, 135-136, 138-145, 147, 149-153, 159-160, 162, 164-165, 167-173, 175-177, 179-195, 197-203, 205-241, 243-249, 252-253, 255-257, 259-269, 271-273, 275, 277-282, 284-288, 290-302, 304-313, 316-320, 322-329, 331-350, 352-356, 358-359, 361-366, 368-371, 373-374, 376, 378-381, 383-399, 401, 403-425, 427-442, 444-445, 447-449, 451-456, 458-463, 467-469, 471-474, 476-477, 481-490, 493, 495-497, 499-579, 581-588, 590-599, 601-613, 615-616, 618-628, 630-634, 637, 639-641, 643-645, 647-658, 660-681, 683-719, 722, 724-727, 729-733, 735-743, 745-747, 749-751, 753-759, 761-765, 767-769, 771, 773-775, 777-787, 789-790, 792-816, 818-830, 832-835, 837, 839-854, 856-859, 861-865, 867-869, 871-872, 874-879, 881-883, 885-899, 901-905, 907-919, 922-925, 927-941, 943-946, 948-950, 952-961, 963-1001, 1003-1004, 1007, 1009, 1011-1023]
	stream status: stopped
	len: 124
	vb list dump: [1, 7, 11, 14, 27, 31, 38, 66, 75, 85, 105, 108, 113, 134, 137, 146, 148, 154-158, 161, 163, 166, 174, 178, 196, 204, 242, 250-251, 254, 258, 270, 274, 276, 283, 289, 303, 314-315, 321, 330, 351, 357, 360, 367, 372, 375, 377, 382, 400, 402, 426, 443, 446, 450, 457, 464-466, 470, 475, 478-480, 491-492, 494, 498, 580, 589, 600, 614, 617, 629, 635-636, 638, 642, 646, 659, 682, 720-721, 723, 728, 734, 744, 748, 752, 760, 766, 770, 772, 776, 788, 791, 817, 831, 836, 838, 855, 860, 866, 870, 873, 880, 884, 900, 906, 920-921, 926, 942, 947, 951, 962, 1002, 1005-1006, 1008, 1010]

vbucket curr owner:
Producer node: 172.31.2.134:9300	Node UUID: e819be24711971bbc2204265433dadcd
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [683-719, 722, 724-727, 729-733, 735-743, 745-747, 749-751, 753-759, 761-765, 767-769, 771, 773-775, 777-787, 789-790, 792-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-816, 818-830, 832-835, 837, 839-854, 856-859, 861-865, 867-869, 871-872, 874-879, 881-883, 885-899, 901-905, 907-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-919, 922-925, 927-941, 943-946, 948-950, 952-961, 963-1001, 1003-1004, 1007, 1009, 1011-1023]
Producer node: 127.0.0.1:9303	Node UUID: af7660ed7963dc5cc343f4149d950c83
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [342-350, 352-356, 358-359, 361-366, 368-371, 373-374, 376, 378-381, 383-399, 401, 403-425, 427-442, 444-445, 447-449, 451-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458-463, 467-469, 471-474, 476-477, 481-490, 493, 495-497, 499-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 100
	v: [570-579, 581-588, 590-599, 601-613, 615-616, 618-628, 630-634, 637, 639-641, 643-645, 647-658, 660-681]
Producer node: 127.0.0.1:9302	Node UUID: 380e3557fb5e287436052c7c7a803448
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 99
	v: [228-241, 243-249, 252-253, 255-257, 259-269, 271-273, 275, 277-282, 284-288, 290-302, 304-313, 316-320, 322-329, 331-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [0, 2-6, 8-10, 12-13, 15-26, 28-30, 32-37, 39-65, 67-74, 76-84, 86-104, 106-107, 109-112]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 98
	v: [114-133, 135-136, 138-145, 147, 149-153, 159-160, 162, 164-165, 167-173, 175-177, 179-195, 197-203, 205-227]


2021/12/13 22:23:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:23:44 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/13 22:23:44 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/13 22:23:49 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/13 22:23:49 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/13 22:23:49 eventingSetup.go:46: No eventing-con process running
2021/12/13 22:23:49 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/13 22:23:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:23:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:23:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:24:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:24:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 22:24:02 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 22:24:07 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (312.80s)
=== RUN   TestN1QLRecursion
2021/12/13 22:24:10 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 22:24:10 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 22:24:10 Removing node(s): [] from the cluster
2021/12/13 22:24:15 Rebalance progress: 100
2021/12/13 22:24:21 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:24:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:24:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:24:38 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:24:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:24:39 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:24:44 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 22:24:49 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 22:24:54 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 22:24:59 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/13 22:24:59 App: TestN1QLRecursion_function1 got deployed
2021/12/13 22:24:59 TestN1QLRecursion_function1 is deployed
2021/12/13 22:24:59 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:25:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 22:25:00 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:25:05 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 22:25:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 22:25:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 22:25:20 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/13 22:25:20 App: TestN1QLRecursion_function3 got deployed
2021/12/13 22:25:20 TestN1QLRecursion_function3 is deployed
2021/12/13 22:25:20 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:25:20 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/13 22:25:20 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/13 22:25:20 Success: TestN1QLRecursion_function2 is not deployed
2021/12/13 22:25:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:25:26 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 22:25:31 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 22:25:36 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 22:25:41 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/13 22:25:41 App: TestN1QLRecursion_function1 got un-deployed
2021/12/13 22:25:41 No eventing-con process running
2021/12/13 22:25:41 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/13 22:25:43 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:25:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:25:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:25:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:25:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:26:00 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 22:26:05 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 22:26:10 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 22:26:15 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/13 22:26:15 App: TestN1QLRecursion_function3 got un-deployed
2021/12/13 22:26:15 No eventing-con process running
2021/12/13 22:26:15 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/13 22:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:26:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:26:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:26:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:26:30 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 22:26:35 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.38s)
=== RUN   TestN1QLAllowRecursion
2021/12/13 22:26:35 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 22:26:36 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 22:26:36 Removing node(s): [] from the cluster
2021/12/13 22:26:41 Rebalance progress: 100
2021/12/13 22:26:46 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:26:53 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:26:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:27:04 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:27:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:27:05 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:27:10 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 22:27:15 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 22:27:20 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 22:27:25 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/13 22:27:25 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/13 22:27:25 TestN1QLAllowRecursion_function1 is deployed
2021/12/13 22:27:25 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:27:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/13 22:27:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:27:31 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 22:27:36 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 22:27:41 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 22:27:46 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/13 22:27:46 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/13 22:27:46 TestN1QLAllowRecursion_function3 is deployed
2021/12/13 22:27:46 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:27:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/13 22:27:47 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/13 22:27:52 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 22:27:57 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 22:28:02 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 22:28:07 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/13 22:28:07 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/13 22:28:07 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/13 22:28:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:28:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 22:28:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 22:28:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 22:28:27 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/13 22:28:27 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/13 22:28:27 No eventing-con process running
2021/12/13 22:28:27 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/13 22:28:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:28:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:28:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:28:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:28:41 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:28:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 22:28:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 22:28:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 22:29:01 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/13 22:29:01 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/13 22:29:01 No eventing-con process running
2021/12/13 22:29:01 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/13 22:29:03 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:29:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:29:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:29:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:29:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:29:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 22:29:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 22:29:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 22:29:35 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/13 22:29:35 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/13 22:29:35 No eventing-con process running
2021/12/13 22:29:35 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/13 22:29:37 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:29:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:29:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:29:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:29:50 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 22:29:56 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.19s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/13 22:29:56 Sleeping for some time
2021/12/13 22:30:06 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 22:30:06 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/13 22:30:06 No eventing-con process running
2021/12/13 22:30:06 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/13 22:30:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:30:14 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:30:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:30:19 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:30:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 22:30:20 Starting bucket ops to source bucket
2021/12/13 22:31:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 22:31:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 22:31:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 22:31:15 Starting bucket ops to source bucket
2021/12/13 22:31:16 Starting bucket ops to source bucket
2021/12/13 22:32:22 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/13 22:32:22 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/13 22:32:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:32:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:32:27 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 22:32:32 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 22:32:37 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 22:32:42 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/13 22:32:42 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/13 22:32:42 No eventing-con process running
2021/12/13 22:32:42 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/13 22:32:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:32:54 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:32:59 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/13 22:32:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:33:00 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:33:10 Waiting for dst bucket item count to get to: 0 curr count: 5790
2021/12/13 22:33:15 Waiting for dst bucket item count to get to: 0 curr count: 5790
2021/12/13 22:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (199.97s)
=== RUN   TestTimerOverWriteSameReference
2021/12/13 22:33:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 22:33:20 App: TestTimerOverWriteSameReference got un-deployed
2021/12/13 22:33:21 No eventing-con process running
2021/12/13 22:33:21 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/13 22:33:22 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:33:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:33:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:33:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:33:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:33:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/13 22:33:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 22:33:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 22:33:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 22:33:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/13 22:33:55 App: TestTimerOverWriteSameReference got deployed
2021/12/13 22:34:05 Starting bucket ops to source bucket
2021/12/13 22:35:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/13 22:35:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:35:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:35:22 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 22:35:27 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 22:35:32 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 22:35:37 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/13 22:35:37 App: TestTimerOverWriteSameReference got un-deployed
2021/12/13 22:35:37 No eventing-con process running
2021/12/13 22:35:37 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/13 22:35:39 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:35:49 Waiting for dst bucket item count to get to: 0 curr count: 854
2021/12/13 22:35:54 Waiting for dst bucket item count to get to: 0 curr count: 354
2021/12/13 22:35:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:35:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:36:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/13 22:36:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.87s)
=== RUN   TestCancelTimerBucketop
2021/12/13 22:36:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 22:36:10 App: TestCancelTimerBucketop got un-deployed
2021/12/13 22:36:10 No eventing-con process running
2021/12/13 22:36:10 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/13 22:36:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:36:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:36:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:36:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:36:24 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:36:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/13 22:36:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 22:36:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 22:36:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 22:36:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/13 22:36:45 App: TestCancelTimerBucketop got deployed
2021/12/13 22:36:45 Starting bucket ops to source bucket
2021/12/13 22:36:52 Starting bucket ops to source bucket
2021/12/13 22:37:55 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 22:37:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:37:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:38:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 22:38:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 22:38:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 22:38:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/13 22:38:15 App: TestCancelTimerBucketop got un-deployed
2021/12/13 22:38:15 No eventing-con process running
2021/12/13 22:38:15 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/13 22:38:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:38:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:38:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:38:38 Waiting for dst bucket item count to get to: 0 curr count: 866
2021/12/13 22:38:43 Waiting for dst bucket item count to get to: 0 curr count: 366
2021/12/13 22:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.10s)
=== RUN   TestJSExpiryDate
2021/12/13 22:38:48 Starting bucket ops to source bucket
2021/12/13 22:39:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 22:39:06 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/13 22:39:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/13 22:39:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:39:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:39:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:39:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 22:39:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 22:39:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 22:39:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/13 22:39:53 App: TestJSExpiryDate got un-deployed
2021/12/13 22:39:53 No eventing-con process running
2021/12/13 22:39:53 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/13 22:39:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:40:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:40:02 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:40:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.64s)
=== RUN   TestBinaryDoc
2021/12/13 22:40:12 Starting bucket ops to source bucket
2021/12/13 22:40:12 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:40:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/13 22:40:13 Starting bucket ops to source bucket
2021/12/13 22:40:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:40:13 Deleting source bucket docs now
2021/12/13 22:40:13 Starting bucket ops to source bucket
2021/12/13 22:40:51 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/13 22:40:56 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/13 22:40:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/13 22:40:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:40:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:41:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 22:41:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 22:41:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 22:41:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/13 22:41:16 App: TestBinaryDoc got un-deployed
2021/12/13 22:41:16 No eventing-con process running
2021/12/13 22:41:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/13 22:41:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:41:28 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/13 22:41:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:41:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:41:39 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/13 22:41:44 Waiting for dst bucket item count to get to: 0 curr count: 16
2021/12/13 22:41:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.25s)
=== RUN   TestConstantBindings
2021/12/13 22:41:44 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:41:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/13 22:41:45 Starting bucket ops to source bucket
2021/12/13 22:41:51 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 22:41:56 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 22:42:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/13 22:42:06 Waiting for dst bucket item count to get to: 6000 curr count: 3377
2021/12/13 22:42:11 Waiting for dst bucket item count to get to: 6000 curr count: 3377
2021/12/13 22:42:11 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/13 22:42:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:42:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:42:16 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 22:42:21 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 22:42:26 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 22:42:31 Waiting for app: TestConstantBindings to get un-deployed
2021/12/13 22:42:31 App: TestConstantBindings got un-deployed
2021/12/13 22:42:31 No eventing-con process running
2021/12/13 22:42:31 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/13 22:42:33 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:42:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:42:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 22:42:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:42:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:43:00 Waiting for dst bucket item count to get to: 0 curr count: 3769
2021/12/13 22:43:05 Waiting for dst bucket item count to get to: 0 curr count: 3769
2021/12/13 22:43:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.64s)
=== RUN   TestCodeInsight
2021/12/13 22:43:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 22:43:10 App: TestCodeInsight got un-deployed
2021/12/13 22:43:10 No eventing-con process running
2021/12/13 22:43:10 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/13 22:43:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:43:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:43:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:43:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:43:24 Starting bucket ops to source bucket
2021/12/13 22:43:24 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:43:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/13 22:43:30 Waiting for app: TestCodeInsight to get deployed
2021/12/13 22:43:35 Waiting for app: TestCodeInsight to get deployed
2021/12/13 22:43:40 Waiting for app: TestCodeInsight to get deployed
2021/12/13 22:43:45 Waiting for app: TestCodeInsight to get deployed
2021/12/13 22:43:45 App: TestCodeInsight got deployed
2021/12/13 22:43:50 Waiting for dst bucket item count to get to: 1000 curr count: 501
2021/12/13 22:43:50 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 22:43:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:43:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 22:44:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 22:44:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 22:44:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/13 22:44:11 App: TestCodeInsight got un-deployed
2021/12/13 22:44:11 No eventing-con process running
2021/12/13 22:44:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/13 22:44:13 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:44:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:44:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 22:44:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:44:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:44:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 22:44:44 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 22:44:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.41s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.05s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12719.488s

Functional tests took 3 hours, 32 minutes and 4 seconds.

N1ql tests

2021/12/13 22:44:54 Started n1ql
2021/12/13 22:45:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/13 22:45:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/13 22:45:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 22:45:05 Removing node(s): [] from the cluster
2021/12/13 22:45:10 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/13 22:45:15 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/13 22:45:15 App: TestRecursiveMutationN1QL got un-deployed
2021/12/13 22:45:15 No eventing-con process running
2021/12/13 22:45:15 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/13 22:45:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:45:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:45:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:45:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:45:29 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:45:29 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.51s)
=== RUN   TestFlexReset1
2021/12/13 22:45:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 22:45:34 App: TestFlexReset1 got un-deployed
2021/12/13 22:45:34 No eventing-con process running
2021/12/13 22:45:34 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/13 22:45:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:45:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:45:42 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:45:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:45:47 Starting bucket ops to source bucket
2021/12/13 22:45:48 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:45:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/13 22:45:54 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 22:45:59 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 22:46:04 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 22:46:09 Waiting for app: TestFlexReset1 to get deployed
2021/12/13 22:46:09 App: TestFlexReset1 got deployed
2021/12/13 22:46:14 Waiting for dst bucket item count to get to: 1000 curr count: 500
2021/12/13 22:46:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 22:46:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:46:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:46:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 22:46:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 22:46:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 22:46:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/13 22:46:34 App: TestFlexReset1 got un-deployed
2021/12/13 22:46:34 No eventing-con process running
2021/12/13 22:46:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/13 22:46:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:46:46 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:46:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:46:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:46:58 Waiting for dst bucket item count to get to: 0 curr count: 686
2021/12/13 22:47:03 Waiting for dst bucket item count to get to: 0 curr count: 186
2021/12/13 22:47:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (94.21s)
=== RUN   TestFlexReset2
2021/12/13 22:47:08 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 22:47:08 App: TestFlexReset2 got un-deployed
2021/12/13 22:47:08 No eventing-con process running
2021/12/13 22:47:08 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/13 22:47:10 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:47:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:47:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:47:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:47:22 Starting bucket ops to source bucket
2021/12/13 22:47:22 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:47:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/13 22:47:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 22:47:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 22:47:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 22:47:43 Waiting for app: TestFlexReset2 to get deployed
2021/12/13 22:47:43 App: TestFlexReset2 got deployed
2021/12/13 22:47:48 Waiting for dst bucket item count to get to: 1000 curr count: 752
2021/12/13 22:47:54 Waiting for dst bucket item count to get to: 1000 curr count: 752
2021/12/13 22:47:54 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/13 22:47:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:47:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:47:59 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 22:48:04 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 22:48:09 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 22:48:14 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/13 22:48:14 App: TestFlexReset2 got un-deployed
2021/12/13 22:48:14 No eventing-con process running
2021/12/13 22:48:14 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/13 22:48:16 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:48:26 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:48:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:48:37 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/13 22:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (94.32s)
=== RUN   TestN1QLLabelledBreak
2021/12/13 22:48:42 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 22:48:42 App: TestN1QLLabelledBreak got un-deployed
2021/12/13 22:48:42 No eventing-con process running
2021/12/13 22:48:42 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/13 22:48:44 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:48:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:48:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:48:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:48:56 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:48:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/13 22:49:02 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 22:49:07 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 22:49:12 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 22:49:17 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/13 22:49:17 App: TestN1QLLabelledBreak got deployed
2021/12/13 22:49:17 Starting bucket ops to source bucket
2021/12/13 22:49:27 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/13 22:49:32 Waiting for dst bucket item count to get to: 10000 curr count: 137
2021/12/13 22:49:37 Waiting for dst bucket item count to get to: 10000 curr count: 137
2021/12/13 22:49:42 Waiting for dst bucket item count to get to: 10000 curr count: 3284
2021/12/13 22:49:47 Waiting for dst bucket item count to get to: 10000 curr count: 3284
2021/12/13 22:49:52 Waiting for dst bucket item count to get to: 10000 curr count: 7384
2021/12/13 22:49:57 Waiting for dst bucket item count to get to: 10000 curr count: 7384
2021/12/13 22:49:57 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 22:49:57 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:49:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:50:02 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 22:50:07 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 22:50:12 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 22:50:17 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/13 22:50:17 App: TestN1QLLabelledBreak got un-deployed
2021/12/13 22:50:17 No eventing-con process running
2021/12/13 22:50:17 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/13 22:50:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:50:29 Waiting for dst bucket item count to get to: 0 curr count: 4552
2021/12/13 22:50:34 Waiting for dst bucket item count to get to: 0 curr count: 2052
2021/12/13 22:50:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:50:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:50:46 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 22:50:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.20s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/13 22:50:51 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 22:50:51 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/13 22:50:51 No eventing-con process running
2021/12/13 22:50:51 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/13 22:50:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:50:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:50:59 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:51:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:51:04 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:51:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/13 22:51:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 22:51:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 22:51:20 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 22:51:25 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/13 22:51:25 App: TestN1QLUnlabelledBreak got deployed
2021/12/13 22:51:25 Starting bucket ops to source bucket
2021/12/13 22:51:35 Waiting for dst bucket item count to get to: 10000 curr count: 600
2021/12/13 22:51:40 Waiting for dst bucket item count to get to: 10000 curr count: 600
2021/12/13 22:51:45 Waiting for dst bucket item count to get to: 10000 curr count: 5726
2021/12/13 22:51:50 Waiting for dst bucket item count to get to: 10000 curr count: 5726
2021/12/13 22:51:55 Waiting for dst bucket item count to get to: 10000 curr count: 5726
2021/12/13 22:51:55 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 22:51:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:51:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:52:00 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 22:52:05 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 22:52:10 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 22:52:15 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/13 22:52:15 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/13 22:52:15 No eventing-con process running
2021/12/13 22:52:15 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/13 22:52:17 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:52:27 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 22:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:52:29 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:52:39 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 22:52:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 22:52:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (118.50s)
=== RUN   TestN1QLThrowStatement
2021/12/13 22:52:49 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 22:52:49 App: TestN1QLThrowStatement got un-deployed
2021/12/13 22:52:49 No eventing-con process running
2021/12/13 22:52:49 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/13 22:52:51 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:52:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:52:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:53:03 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:53:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/13 22:53:09 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 22:53:14 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 22:53:19 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 22:53:24 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/13 22:53:24 App: TestN1QLThrowStatement got deployed
2021/12/13 22:53:24 Starting bucket ops to source bucket
2021/12/13 22:53:33 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/13 22:53:38 Waiting for dst bucket item count to get to: 10000 curr count: 237
2021/12/13 22:53:43 Waiting for dst bucket item count to get to: 10000 curr count: 739
2021/12/13 22:53:48 Waiting for dst bucket item count to get to: 10000 curr count: 2595
2021/12/13 22:53:53 Waiting for dst bucket item count to get to: 10000 curr count: 4622
2021/12/13 22:53:58 Waiting for dst bucket item count to get to: 10000 curr count: 6660
2021/12/13 22:54:03 Waiting for dst bucket item count to get to: 10000 curr count: 8671
2021/12/13 22:54:08 Waiting for dst bucket item count to get to: 10000 curr count: 9540
2021/12/13 22:54:08 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/13 22:54:08 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:54:08 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:54:13 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 22:54:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 22:54:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 22:54:29 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/13 22:54:29 App: TestN1QLThrowStatement got un-deployed
2021/12/13 22:54:29 No eventing-con process running
2021/12/13 22:54:29 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/13 22:54:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:54:40 Waiting for dst bucket item count to get to: 0 curr count: 3723
2021/12/13 22:54:45 Waiting for dst bucket item count to get to: 0 curr count: 3723
2021/12/13 22:54:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:54:47 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:54:57 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/13 22:54:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.10s)
=== RUN   TestN1QLNestedForLoop
2021/12/13 22:55:02 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 22:55:02 App: TestN1QLNestedForLoop got un-deployed
2021/12/13 22:55:02 No eventing-con process running
2021/12/13 22:55:02 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/13 22:55:04 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:55:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:55:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:55:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:55:16 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:55:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/13 22:55:22 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 22:55:27 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 22:55:32 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 22:55:37 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/13 22:55:37 App: TestN1QLNestedForLoop got deployed
2021/12/13 22:55:37 Starting bucket ops to source bucket
2021/12/13 22:55:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 22:55:51 Waiting for dst bucket item count to get to: 5000 curr count: 80
2021/12/13 22:55:56 Waiting for dst bucket item count to get to: 5000 curr count: 80
2021/12/13 22:56:01 Waiting for dst bucket item count to get to: 5000 curr count: 1945
2021/12/13 22:56:06 Waiting for dst bucket item count to get to: 5000 curr count: 1945
2021/12/13 22:56:11 Waiting for dst bucket item count to get to: 5000 curr count: 4334
2021/12/13 22:56:16 Waiting for dst bucket item count to get to: 5000 curr count: 4334
2021/12/13 22:56:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 22:56:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 22:56:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 22:56:21 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 22:56:27 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 22:56:32 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 22:56:37 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/13 22:56:37 App: TestN1QLNestedForLoop got un-deployed
2021/12/13 22:56:37 No eventing-con process running
2021/12/13 22:56:37 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/13 22:56:38 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:56:48 Waiting for dst bucket item count to get to: 0 curr count: 3944
2021/12/13 22:56:53 Waiting for dst bucket item count to get to: 0 curr count: 1444
2021/12/13 22:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:56:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:57:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 22:57:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.07s)
=== RUN   TestN1QLAutoClose
2021/12/13 22:57:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 22:57:10 App: TestN1QLAutoClose got un-deployed
2021/12/13 22:57:10 No eventing-con process running
2021/12/13 22:57:10 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/13 22:57:12 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:57:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:57:19 Context: Bucket flush, response code: 200 dump: 
2021/12/13 22:57:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 22:57:24 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/13 22:57:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/13 22:57:30 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 22:57:35 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 22:57:40 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 22:57:45 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/13 22:57:45 App: TestN1QLAutoClose got deployed
2021/12/13 22:57:45 Starting bucket ops to source bucket
2021/12/13 22:57:50 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/13 22:57:56 Waiting for dst bucket item count to get to: 500 curr count: 2
2021/12/13 22:58:01 Waiting for dst bucket item count to get to: 500 curr count: 5
2021/12/13 22:58:06 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/13 22:58:11 Waiting for dst bucket item count to get to: 500 curr count: 28
2021/12/13 22:58:16 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/13 22:58:21 Waiting for dst bucket item count to get to: 500 curr count: 47
2021/12/13 22:58:26 Waiting for dst bucket item count to get to: 500 curr count: 71
2021/12/13 22:58:31 Waiting for dst bucket item count to get to: 500 curr count: 71
2021/12/13 22:58:36 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/13 22:58:41 Waiting for dst bucket item count to get to: 500 curr count: 92
2021/12/13 22:58:46 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/13 22:58:51 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/13 22:58:56 Waiting for dst bucket item count to get to: 500 curr count: 112
2021/12/13 22:59:01 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/13 22:59:06 Waiting for dst bucket item count to get to: 500 curr count: 134
2021/12/13 22:59:11 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/13 22:59:16 Waiting for dst bucket item count to get to: 500 curr count: 155
2021/12/13 22:59:21 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/13 22:59:26 Waiting for dst bucket item count to get to: 500 curr count: 176
2021/12/13 22:59:31 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/13 22:59:36 Waiting for dst bucket item count to get to: 500 curr count: 199
2021/12/13 22:59:41 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/13 22:59:46 Waiting for dst bucket item count to get to: 500 curr count: 221
2021/12/13 22:59:51 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/13 22:59:56 Waiting for dst bucket item count to get to: 500 curr count: 240
2021/12/13 23:00:01 Waiting for dst bucket item count to get to: 500 curr count: 261
2021/12/13 23:00:06 Waiting for dst bucket item count to get to: 500 curr count: 261
2021/12/13 23:00:11 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/13 23:00:16 Waiting for dst bucket item count to get to: 500 curr count: 283
2021/12/13 23:00:21 Waiting for dst bucket item count to get to: 500 curr count: 305
2021/12/13 23:00:26 Waiting for dst bucket item count to get to: 500 curr count: 318
2021/12/13 23:00:31 Waiting for dst bucket item count to get to: 500 curr count: 327
2021/12/13 23:00:36 Waiting for dst bucket item count to get to: 500 curr count: 338
2021/12/13 23:00:41 Waiting for dst bucket item count to get to: 500 curr count: 347
2021/12/13 23:00:46 Waiting for dst bucket item count to get to: 500 curr count: 358
2021/12/13 23:00:51 Waiting for dst bucket item count to get to: 500 curr count: 368
2021/12/13 23:00:56 Waiting for dst bucket item count to get to: 500 curr count: 377
2021/12/13 23:01:01 Waiting for dst bucket item count to get to: 500 curr count: 386
2021/12/13 23:01:06 Waiting for dst bucket item count to get to: 500 curr count: 397
2021/12/13 23:01:11 Waiting for dst bucket item count to get to: 500 curr count: 408
2021/12/13 23:01:16 Waiting for dst bucket item count to get to: 500 curr count: 421
2021/12/13 23:01:21 Waiting for dst bucket item count to get to: 500 curr count: 430
2021/12/13 23:01:26 Waiting for dst bucket item count to get to: 500 curr count: 442
2021/12/13 23:01:31 Waiting for dst bucket item count to get to: 500 curr count: 452
2021/12/13 23:01:36 Waiting for dst bucket item count to get to: 500 curr count: 463
2021/12/13 23:01:41 Waiting for dst bucket item count to get to: 500 curr count: 473
2021/12/13 23:01:46 Waiting for dst bucket item count to get to: 500 curr count: 482
2021/12/13 23:01:51 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/13 23:01:56 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/13 23:01:56 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/13 23:01:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 23:01:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:02:01 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 23:02:06 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 23:02:11 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 23:02:16 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/13 23:02:16 App: TestN1QLAutoClose got un-deployed
2021/12/13 23:02:16 No eventing-con process running
2021/12/13 23:02:16 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/13 23:02:18 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:02:28 Waiting for dst bucket item count to get to: 0 curr count: 306
2021/12/13 23:02:33 Waiting for dst bucket item count to get to: 0 curr count: 56
2021/12/13 23:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:02:35 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:02:45 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/13 23:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.54s)
=== RUN   TestN1QLPosParams
2021/12/13 23:02:50 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 23:02:50 App: TestN1QLPosParams got un-deployed
2021/12/13 23:02:50 No eventing-con process running
2021/12/13 23:02:50 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/13 23:02:52 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:02:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:02:58 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:03:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:03:03 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:03:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/13 23:03:10 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 23:03:15 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 23:03:20 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 23:03:25 Waiting for app: TestN1QLPosParams to get deployed
2021/12/13 23:03:25 App: TestN1QLPosParams got deployed
2021/12/13 23:03:25 Starting bucket ops to source bucket
2021/12/13 23:03:34 Waiting for dst bucket item count to get to: 5000 curr count: 412
2021/12/13 23:03:39 Waiting for dst bucket item count to get to: 5000 curr count: 412
2021/12/13 23:03:44 Waiting for dst bucket item count to get to: 5000 curr count: 1223
2021/12/13 23:03:49 Waiting for dst bucket item count to get to: 5000 curr count: 3311
2021/12/13 23:03:49 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 23:03:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 23:03:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:03:54 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 23:03:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 23:04:04 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 23:04:09 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/13 23:04:09 App: TestN1QLPosParams got un-deployed
2021/12/13 23:04:09 No eventing-con process running
2021/12/13 23:04:09 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/13 23:04:11 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:04:21 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 23:04:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 23:04:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:04:28 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:04:38 Waiting for dst bucket item count to get to: 0 curr count: 3333
2021/12/13 23:04:43 Waiting for dst bucket item count to get to: 0 curr count: 833
2021/12/13 23:04:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (118.21s)
=== RUN   TestN1QLExhaustConnPool
2021/12/13 23:04:48 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 23:04:48 App: TestN1QLExhaustConnPool got un-deployed
2021/12/13 23:04:48 No eventing-con process running
2021/12/13 23:04:48 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/13 23:04:50 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:04:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:04:57 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:05:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:05:02 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:05:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/13 23:05:08 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 23:05:13 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 23:05:18 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 23:05:23 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/13 23:05:23 App: TestN1QLExhaustConnPool got deployed
2021/12/13 23:05:23 Starting bucket ops to source bucket
2021/12/13 23:05:28 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 23:05:33 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/13 23:05:33 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/13 23:05:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 23:05:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:05:38 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 23:05:43 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 23:05:48 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 23:05:53 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/13 23:05:53 App: TestN1QLExhaustConnPool got un-deployed
2021/12/13 23:05:53 No eventing-con process running
2021/12/13 23:05:53 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/13 23:05:55 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:06:05 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 23:06:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:06:07 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:06:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/13 23:06:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (93.82s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/13 23:06:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 23:06:22 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/13 23:06:22 No eventing-con process running
2021/12/13 23:06:22 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/13 23:06:24 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:06:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:06:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:06:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:06:35 Starting bucket ops to source bucket
2021/12/13 23:06:53 Starting bucket ops to source bucket
2021/12/13 23:07:11 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 8846
2021/12/13 23:07:12 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/13 23:07:17 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 23:07:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 23:07:27 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 23:07:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/13 23:07:32 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/13 23:07:32 Starting bucket ops to source bucket
2021/12/13 23:07:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:07:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:07:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:07:52 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:07:57 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:02 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:07 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:12 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:17 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:22 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/13 23:08:27 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/13 23:08:27 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 23:08:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:08:32 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 23:08:37 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 23:08:42 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 23:08:47 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/13 23:08:47 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/13 23:08:47 No eventing-con process running
2021/12/13 23:08:47 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/13 23:08:49 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:08:59 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/13 23:09:04 Waiting for dst bucket item count to get to: 0 curr count: 15000
2021/12/13 23:09:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:09:06 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:09:16 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/13 23:09:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (179.27s)
=== RUN   TestN1QLQueryConsistency
2021/12/13 23:09:21 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 23:09:21 App: n1qlquery_consistency got un-deployed
2021/12/13 23:09:21 No eventing-con process running
2021/12/13 23:09:21 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/13 23:09:23 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:09:30 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:09:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:09:35 Starting bucket ops to source bucket
2021/12/13 23:09:38 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:09:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/13 23:09:44 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 23:09:49 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 23:09:54 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 23:09:59 Waiting for app: n1qlquery_consistency to get deployed
2021/12/13 23:09:59 App: n1qlquery_consistency got deployed
2021/12/13 23:10:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/13 23:10:09 Waiting for dst bucket item count to get to: 5000 curr count: 4390
2021/12/13 23:10:14 Waiting for dst bucket item count to get to: 5000 curr count: 4390
2021/12/13 23:10:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/13 23:10:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/13 23:10:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:10:19 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 23:10:24 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 23:10:29 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 23:10:34 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/13 23:10:34 App: n1qlquery_consistency got un-deployed
2021/12/13 23:10:34 No eventing-con process running
2021/12/13 23:10:34 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/13 23:10:36 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:10:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/13 23:10:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:10:48 Context: Bucket flush, response code: 200 dump: 
2021/12/13 23:10:58 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/13 23:10:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (101.59s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1559.487s

N1ql tests took 0 hours, 26 minutes and 4 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/13 23:12:33 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/13 23:12:50 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
running apps fetch error json: cannot unmarshal number into Go value of type string
2021/12/13 23:12:55 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 23:12:55 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/13 23:12:55 No eventing-con process running
2021/12/13 23:12:55 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/13 23:12:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:13:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:13:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:13:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:13:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:13:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:13:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:13:14 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:13:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 23:13:20 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 23:13:25 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 23:13:30 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 23:13:35 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/13 23:13:35 App: TestDuplicateWithEventingRebIn got deployed
2021/12/13 23:13:35 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:13:35 Starting bucket ops to source bucket
2021/12/13 23:14:03 Removing node(s): [] from the cluster
2021/12/13 23:14:08 Rebalance progress: 0
2021/12/13 23:14:13 Rebalance progress: 0
2021/12/13 23:14:18 Rebalance progress: 25
2021/12/13 23:14:23 Rebalance progress: 25
2021/12/13 23:14:28 Rebalance progress: 25
2021/12/13 23:14:33 Rebalance progress: 98.33333333333334
2021/12/13 23:14:38 Rebalance progress: 98.33333333333334
2021/12/13 23:14:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:14:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 23:14:43 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:14:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:14:48 Rebalance progress: 25
2021/12/13 23:14:53 Rebalance progress: 25
2021/12/13 23:14:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:14:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:14:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 23:14:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:14:58 Waiting for src bucket item count to get to: 200000 curr count: 121665
2021/12/13 23:15:03 Waiting for src bucket item count to get to: 200000 curr count: 138186
2021/12/13 23:15:08 Waiting for src bucket item count to get to: 200000 curr count: 138186
2021/12/13 23:15:13 Waiting for src bucket item count to get to: 200000 curr count: 154721
2021/12/13 23:15:18 Waiting for src bucket item count to get to: 200000 curr count: 154721
2021/12/13 23:15:23 Waiting for src bucket item count to get to: 200000 curr count: 171237
2021/12/13 23:15:28 Waiting for src bucket item count to get to: 200000 curr count: 171237
2021/12/13 23:15:33 Waiting for src bucket item count to get to: 200000 curr count: 187490
2021/12/13 23:15:38 Waiting for src bucket item count to get to: 200000 curr count: 187490
2021/12/13 23:15:48 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:15:53 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:15:58 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:03 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:08 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:13 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:18 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:23 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:28 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:33 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/13 23:16:33 Post SET ops. Expected item count: 200000 got 200001
2021/12/13 23:16:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:16:38 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 23:16:43 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 23:16:48 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 23:16:53 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/13 23:16:53 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/13 23:16:53 No eventing-con process running
2021/12/13 23:16:53 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/13 23:16:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:17:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:17:05 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:17:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:17:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:17:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:17:17 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/13 23:17:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (276.50s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/13 23:17:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:17:22 Removing node(s): [] from the cluster
2021/12/13 23:17:27 Rebalance progress: 100
2021/12/13 23:17:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 23:17:32 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/13 23:17:32 No eventing-con process running
2021/12/13 23:17:32 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/13 23:17:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:17:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:17:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:17:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:17:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:17:51 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:17:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 23:17:57 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 23:18:02 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 23:18:07 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 23:18:12 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/13 23:18:12 App: TestDuplicateWithEventingRebOut got deployed
2021/12/13 23:18:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:18:12 Starting bucket ops to source bucket
2021/12/13 23:18:17 Rebalance progress: 25
2021/12/13 23:18:22 Rebalance progress: 25
2021/12/13 23:18:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:18:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:18:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:18:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 23:18:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:18:27 Waiting for src bucket item count to get to: 200000 curr count: 11807
2021/12/13 23:18:32 Waiting for src bucket item count to get to: 200000 curr count: 28091
2021/12/13 23:18:37 Waiting for src bucket item count to get to: 200000 curr count: 28091
2021/12/13 23:18:42 Waiting for src bucket item count to get to: 200000 curr count: 44408
2021/12/13 23:18:47 Waiting for src bucket item count to get to: 200000 curr count: 44408
2021/12/13 23:18:52 Waiting for src bucket item count to get to: 200000 curr count: 60772
2021/12/13 23:18:57 Waiting for src bucket item count to get to: 200000 curr count: 60772
2021/12/13 23:19:02 Waiting for src bucket item count to get to: 200000 curr count: 60772
2021/12/13 23:19:07 Waiting for src bucket item count to get to: 200000 curr count: 76476
2021/12/13 23:19:12 Waiting for src bucket item count to get to: 200000 curr count: 76476
2021/12/13 23:19:17 Waiting for src bucket item count to get to: 200000 curr count: 92309
2021/12/13 23:19:22 Waiting for src bucket item count to get to: 200000 curr count: 92309
2021/12/13 23:19:27 Waiting for src bucket item count to get to: 200000 curr count: 108519
2021/12/13 23:19:32 Waiting for src bucket item count to get to: 200000 curr count: 108519
2021/12/13 23:19:37 Waiting for src bucket item count to get to: 200000 curr count: 124759
2021/12/13 23:19:42 Waiting for src bucket item count to get to: 200000 curr count: 124759
2021/12/13 23:19:47 Waiting for src bucket item count to get to: 200000 curr count: 141005
2021/12/13 23:19:52 Waiting for src bucket item count to get to: 200000 curr count: 141005
2021/12/13 23:19:57 Waiting for src bucket item count to get to: 200000 curr count: 157258
2021/12/13 23:20:02 Waiting for src bucket item count to get to: 200000 curr count: 157258
2021/12/13 23:20:07 Waiting for src bucket item count to get to: 200000 curr count: 172979
2021/12/13 23:20:12 Waiting for src bucket item count to get to: 200000 curr count: 172979
2021/12/13 23:20:17 Waiting for src bucket item count to get to: 200000 curr count: 188891
2021/12/13 23:20:22 Waiting for src bucket item count to get to: 200000 curr count: 188891
2021/12/13 23:20:27 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 23:20:27 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 23:20:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:20:32 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 23:20:37 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 23:20:42 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 23:20:47 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/13 23:20:47 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/13 23:20:47 No eventing-con process running
2021/12/13 23:20:47 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/13 23:20:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:20:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:20:59 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:20:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:21:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:21:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:21:11 Waiting for dst bucket item count to get to: 0 curr count: 135705
2021/12/13 23:21:16 Waiting for dst bucket item count to get to: 0 curr count: 135705
2021/12/13 23:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (238.75s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/13 23:21:26 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 23:21:26 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/13 23:21:26 No eventing-con process running
2021/12/13 23:21:26 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/13 23:21:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:21:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:21:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:21:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:21:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:21:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:21:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:21:44 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:21:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 23:21:50 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 23:21:55 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 23:22:00 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 23:22:05 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/13 23:22:05 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/13 23:22:05 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:22:05 Starting bucket ops to source bucket
2021/12/13 23:22:05 Removing node(s): [] from the cluster
2021/12/13 23:22:11 Rebalance progress: 25
2021/12/13 23:22:16 Rebalance progress: 25
2021/12/13 23:22:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:22:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:22:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/13 23:22:21 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:22:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:22:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:22:33 Rebalance progress: 16.66666666666667
2021/12/13 23:22:38 Rebalance progress: 16.66666666666667
2021/12/13 23:22:43 Rebalance progress: 16.66666666666667
2021/12/13 23:22:48 Rebalance progress: 29.36802973977695
2021/12/13 23:22:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:22:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:22:53 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:22:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:22:58 Rebalance progress: 16.66666666666667
2021/12/13 23:23:03 Rebalance progress: 16.66666666666667
2021/12/13 23:23:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:23:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:23:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:23:08 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:23:08 Waiting for src bucket item count to get to: 200000 curr count: 81086
2021/12/13 23:23:13 Waiting for src bucket item count to get to: 200000 curr count: 96895
2021/12/13 23:23:18 Waiting for src bucket item count to get to: 200000 curr count: 96895
2021/12/13 23:23:23 Waiting for src bucket item count to get to: 200000 curr count: 113459
2021/12/13 23:23:28 Waiting for src bucket item count to get to: 200000 curr count: 113459
2021/12/13 23:23:33 Waiting for src bucket item count to get to: 200000 curr count: 129687
2021/12/13 23:23:38 Waiting for src bucket item count to get to: 200000 curr count: 129687
2021/12/13 23:23:43 Waiting for src bucket item count to get to: 200000 curr count: 145927
2021/12/13 23:23:48 Waiting for src bucket item count to get to: 200000 curr count: 145927
2021/12/13 23:23:53 Waiting for src bucket item count to get to: 200000 curr count: 162258
2021/12/13 23:23:58 Waiting for src bucket item count to get to: 200000 curr count: 162258
2021/12/13 23:24:03 Waiting for src bucket item count to get to: 200000 curr count: 177857
2021/12/13 23:24:08 Waiting for src bucket item count to get to: 200000 curr count: 177857
2021/12/13 23:24:13 Waiting for src bucket item count to get to: 200000 curr count: 194293
2021/12/13 23:24:18 Waiting for src bucket item count to get to: 200000 curr count: 194293
2021/12/13 23:24:23 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 23:24:23 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 23:24:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:24:28 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 23:24:33 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 23:24:38 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 23:24:43 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/13 23:24:43 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/13 23:24:43 No eventing-con process running
2021/12/13 23:24:43 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/13 23:24:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:24:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:24:55 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:24:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:24:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:25:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:25:07 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:25:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.97s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/13 23:25:17 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 23:25:17 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/13 23:25:17 No eventing-con process running
2021/12/13 23:25:17 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/13 23:25:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:25:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:25:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:25:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:25:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:25:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:25:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:25:35 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:25:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/13 23:25:41 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 23:25:46 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 23:25:51 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 23:25:56 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/13 23:25:56 App: TestDuplicateWithKVRebIn got deployed
2021/12/13 23:25:56 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 23:25:56 Starting bucket ops to source bucket
2021/12/13 23:25:57 Removing node(s): [] from the cluster
2021/12/13 23:26:02 Rebalance progress: 16.66666666666667
2021/12/13 23:26:07 Rebalance progress: 16.66666666666667
2021/12/13 23:26:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:26:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:26:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:26:12 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:26:17 Rebalance progress: 16.66666666666667
2021/12/13 23:26:22 Rebalance progress: 16.66666666666667
2021/12/13 23:26:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:26:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:26:27 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:26:27 Waiting for src bucket item count to get to: 200000 curr count: 35778
2021/12/13 23:26:32 Waiting for src bucket item count to get to: 200000 curr count: 51925
2021/12/13 23:26:37 Waiting for src bucket item count to get to: 200000 curr count: 51925
2021/12/13 23:26:42 Waiting for src bucket item count to get to: 200000 curr count: 68377
2021/12/13 23:26:47 Waiting for src bucket item count to get to: 200000 curr count: 68377
2021/12/13 23:26:52 Waiting for src bucket item count to get to: 200000 curr count: 83889
2021/12/13 23:26:57 Waiting for src bucket item count to get to: 200000 curr count: 83889
2021/12/13 23:27:02 Waiting for src bucket item count to get to: 200000 curr count: 100263
2021/12/13 23:27:07 Waiting for src bucket item count to get to: 200000 curr count: 100263
2021/12/13 23:27:12 Waiting for src bucket item count to get to: 200000 curr count: 100263
2021/12/13 23:27:17 Waiting for src bucket item count to get to: 200000 curr count: 116401
2021/12/13 23:27:22 Waiting for src bucket item count to get to: 200000 curr count: 116401
2021/12/13 23:27:27 Waiting for src bucket item count to get to: 200000 curr count: 132234
2021/12/13 23:27:32 Waiting for src bucket item count to get to: 200000 curr count: 132234
2021/12/13 23:27:37 Waiting for src bucket item count to get to: 200000 curr count: 148589
2021/12/13 23:27:42 Waiting for src bucket item count to get to: 200000 curr count: 148589
2021/12/13 23:27:47 Waiting for src bucket item count to get to: 200000 curr count: 164820
2021/12/13 23:27:52 Waiting for src bucket item count to get to: 200000 curr count: 164820
2021/12/13 23:27:57 Waiting for src bucket item count to get to: 200000 curr count: 179581
2021/12/13 23:28:02 Waiting for src bucket item count to get to: 200000 curr count: 179581
2021/12/13 23:28:07 Waiting for src bucket item count to get to: 200000 curr count: 195521
2021/12/13 23:28:12 Waiting for src bucket item count to get to: 200000 curr count: 195521
2021/12/13 23:28:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 23:28:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 23:28:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:28:22 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 23:28:27 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 23:28:32 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 23:28:37 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/13 23:28:37 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/13 23:28:37 No eventing-con process running
2021/12/13 23:28:37 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/13 23:28:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:28:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:28:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:28:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:28:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:28:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:29:01 Waiting for dst bucket item count to get to: 0 curr count: 148220
2021/12/13 23:29:06 Waiting for dst bucket item count to get to: 0 curr count: 148220
2021/12/13 23:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.14s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/13 23:29:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 23:29:11 Removing node(s): [] from the cluster
2021/12/13 23:29:16 Rebalance progress: 100
2021/12/13 23:29:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 23:29:21 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/13 23:29:21 No eventing-con process running
2021/12/13 23:29:21 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/13 23:29:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:29:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:29:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:29:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:29:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:29:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:29:40 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:29:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/13 23:29:46 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 23:29:51 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 23:29:56 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 23:30:01 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/13 23:30:01 App: TestDuplicateWithKVRebOut got deployed
2021/12/13 23:30:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:30:01 Starting bucket ops to source bucket
2021/12/13 23:30:06 Rebalance progress: 16.66666666666667
2021/12/13 23:30:11 Rebalance progress: 16.66666666666667
2021/12/13 23:30:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:30:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:30:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:30:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:30:16 Waiting for src bucket item count to get to: 200000 curr count: 13060
2021/12/13 23:30:21 Waiting for src bucket item count to get to: 200000 curr count: 29227
2021/12/13 23:30:26 Waiting for src bucket item count to get to: 200000 curr count: 29227
2021/12/13 23:30:31 Waiting for src bucket item count to get to: 200000 curr count: 45645
2021/12/13 23:30:36 Waiting for src bucket item count to get to: 200000 curr count: 45645
2021/12/13 23:30:41 Waiting for src bucket item count to get to: 200000 curr count: 61993
2021/12/13 23:30:46 Waiting for src bucket item count to get to: 200000 curr count: 61993
2021/12/13 23:30:51 Waiting for src bucket item count to get to: 200000 curr count: 61993
2021/12/13 23:30:56 Waiting for src bucket item count to get to: 200000 curr count: 77641
2021/12/13 23:31:01 Waiting for src bucket item count to get to: 200000 curr count: 77641
2021/12/13 23:31:06 Waiting for src bucket item count to get to: 200000 curr count: 93771
2021/12/13 23:31:11 Waiting for src bucket item count to get to: 200000 curr count: 93771
2021/12/13 23:31:16 Waiting for src bucket item count to get to: 200000 curr count: 110061
2021/12/13 23:31:21 Waiting for src bucket item count to get to: 200000 curr count: 110061
2021/12/13 23:31:26 Waiting for src bucket item count to get to: 200000 curr count: 126514
2021/12/13 23:31:31 Waiting for src bucket item count to get to: 200000 curr count: 126514
2021/12/13 23:31:36 Waiting for src bucket item count to get to: 200000 curr count: 142854
2021/12/13 23:31:41 Waiting for src bucket item count to get to: 200000 curr count: 142854
2021/12/13 23:31:46 Waiting for src bucket item count to get to: 200000 curr count: 159161
2021/12/13 23:31:51 Waiting for src bucket item count to get to: 200000 curr count: 159161
2021/12/13 23:31:56 Waiting for src bucket item count to get to: 200000 curr count: 174903
2021/12/13 23:32:01 Waiting for src bucket item count to get to: 200000 curr count: 174903
2021/12/13 23:32:06 Waiting for src bucket item count to get to: 200000 curr count: 191021
2021/12/13 23:32:11 Waiting for src bucket item count to get to: 200000 curr count: 191021
2021/12/13 23:32:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 23:32:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 23:32:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:32:21 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 23:32:26 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 23:32:31 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 23:32:36 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/13 23:32:36 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/13 23:32:36 No eventing-con process running
2021/12/13 23:32:36 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/13 23:32:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:32:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:32:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:32:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:32:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:32:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.05s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/13 23:33:05 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 23:33:05 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/13 23:33:05 No eventing-con process running
2021/12/13 23:33:05 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/13 23:33:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:33:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:33:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:33:13 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:33:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:33:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:33:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:33:23 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:33:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/13 23:33:29 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 23:33:34 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 23:33:40 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 23:33:45 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/13 23:33:45 App: TestDuplicateWithKVSwapReb got deployed
2021/12/13 23:33:45 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/13 23:33:45 Starting bucket ops to source bucket
2021/12/13 23:33:45 Removing node(s): [] from the cluster
2021/12/13 23:33:50 Rebalance progress: 16.66666666666667
2021/12/13 23:33:55 Rebalance progress: 16.66666666666667
2021/12/13 23:34:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:34:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:34:00 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/13 23:34:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:34:05 Rebalance progress: 16.66666666666667
2021/12/13 23:34:10 Rebalance progress: 16.66666666666667
2021/12/13 23:34:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:34:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:34:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:34:20 Rebalance progress: 16.66666666666667
2021/12/13 23:34:25 Rebalance progress: 16.66666666666667
2021/12/13 23:34:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:34:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:34:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:34:31 Waiting for src bucket item count to get to: 200000 curr count: 69371
2021/12/13 23:34:36 Waiting for src bucket item count to get to: 200000 curr count: 69371
2021/12/13 23:34:41 Waiting for src bucket item count to get to: 200000 curr count: 84033
2021/12/13 23:34:46 Waiting for src bucket item count to get to: 200000 curr count: 84033
2021/12/13 23:34:51 Waiting for src bucket item count to get to: 200000 curr count: 100196
2021/12/13 23:34:56 Waiting for src bucket item count to get to: 200000 curr count: 100196
2021/12/13 23:35:01 Waiting for src bucket item count to get to: 200000 curr count: 116397
2021/12/13 23:35:06 Waiting for src bucket item count to get to: 200000 curr count: 116397
2021/12/13 23:35:11 Waiting for src bucket item count to get to: 200000 curr count: 132575
2021/12/13 23:35:16 Waiting for src bucket item count to get to: 200000 curr count: 132575
2021/12/13 23:35:21 Waiting for src bucket item count to get to: 200000 curr count: 148917
2021/12/13 23:35:26 Waiting for src bucket item count to get to: 200000 curr count: 148917
2021/12/13 23:35:31 Waiting for src bucket item count to get to: 200000 curr count: 148917
2021/12/13 23:35:36 Waiting for src bucket item count to get to: 200000 curr count: 165183
2021/12/13 23:35:41 Waiting for src bucket item count to get to: 200000 curr count: 165183
2021/12/13 23:35:46 Waiting for src bucket item count to get to: 200000 curr count: 180170
2021/12/13 23:35:51 Waiting for src bucket item count to get to: 200000 curr count: 180170
2021/12/13 23:35:56 Waiting for src bucket item count to get to: 200000 curr count: 196146
2021/12/13 23:36:01 Waiting for src bucket item count to get to: 200000 curr count: 196146
2021/12/13 23:36:06 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/13 23:36:06 Post SET ops. Expected item count: 200000 got 200000
2021/12/13 23:36:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:36:11 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 23:36:16 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 23:36:21 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 23:36:26 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/13 23:36:26 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/13 23:36:26 No eventing-con process running
2021/12/13 23:36:26 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/13 23:36:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:36:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:36:38 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:36:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:36:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:36:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:36:49 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/13 23:36:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (234.52s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/13 23:36:59 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 23:36:59 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 23:36:59 No eventing-con process running
2021/12/13 23:36:59 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/13 23:37:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:37:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:37:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:37:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:37:18 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:37:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/13 23:37:24 Starting bucket ops to source bucket
2021/12/13 23:37:29 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 23:37:34 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 23:37:39 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/13 23:37:39 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:37:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:37:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:37:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:37:45 Rebalance progress: 16.66666666666667
2021/12/13 23:37:50 Rebalance progress: 16.66666666666667
2021/12/13 23:37:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:37:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:37:55 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:37:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:37:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:38:00 Rebalance progress: 16.66666666666667
2021/12/13 23:38:05 Rebalance progress: 16.66666666666667
2021/12/13 23:38:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:38:10 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/13 23:38:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 23:38:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:38:20 Rebalance progress: 12.5
2021/12/13 23:38:25 Rebalance progress: 12.5
2021/12/13 23:38:30 Rebalance progress: 12.5
2021/12/13 23:38:35 Rebalance progress: 98.54166666666666
2021/12/13 23:38:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:38:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:38:46 Rebalance progress: 12.5
2021/12/13 23:38:51 Rebalance progress: 12.5
2021/12/13 23:38:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:38:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:38:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:39:01 Rebalance progress: 12.5
2021/12/13 23:39:06 Rebalance progress: 12.5
2021/12/13 23:39:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:39:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:39:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 23:39:16 Rebalance progress: 12.5
2021/12/13 23:39:21 Rebalance progress: 12.5
2021/12/13 23:39:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:39:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:39:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:39:31 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 23:39:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 23:39:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 23:39:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/13 23:39:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/13 23:39:46 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:39:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/13 23:39:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:39:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:39:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12159
2021/12/13 23:39:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:40:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:40:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (195.27s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/13 23:40:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 23:40:15 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/13 23:40:15 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:40:15 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/13 23:40:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:40:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:40:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:40:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:40:33 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:40:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/13 23:40:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:40:44 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 23:40:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 23:40:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/13 23:40:54 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:40:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:40:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:40:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:40:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 23:40:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:41:01 Rebalance progress: 12.5
2021/12/13 23:41:06 Rebalance progress: 12.5
2021/12/13 23:41:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:41:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:41:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:41:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 23:41:16 Rebalance progress: 12.5
2021/12/13 23:41:21 Rebalance progress: 12.5
2021/12/13 23:41:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:41:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:41:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:41:31 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 23:41:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 23:41:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 23:41:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/13 23:41:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/13 23:41:46 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:41:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/13 23:41:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:41:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:41:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4629
2021/12/13 23:41:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:41:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:42:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (119.92s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/13 23:42:15 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 23:42:15 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 23:42:15 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:42:15 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/13 23:42:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:42:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:42:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:42:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:42:33 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:42:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/13 23:42:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:42:44 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 23:42:49 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 23:42:54 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/13 23:42:54 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:42:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:42:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:42:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:43:00 Rebalance progress: 12.5
2021/12/13 23:43:05 Rebalance progress: 12.5
2021/12/13 23:43:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:43:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:43:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:43:15 Rebalance progress: 12.5
2021/12/13 23:43:20 Rebalance progress: 12.5
2021/12/13 23:43:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:43:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 23:43:25 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:43:30 Rebalance progress: 12.5
2021/12/13 23:43:35 Rebalance progress: 12.5
2021/12/13 23:43:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:43:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:43:45 Rebalance progress: 12.5
2021/12/13 23:43:50 Rebalance progress: 12.5
2021/12/13 23:43:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:43:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:43:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:44:00 Rebalance progress: 12.5
2021/12/13 23:44:05 Rebalance progress: 12.5
2021/12/13 23:44:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:44:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 23:44:15 Rebalance progress: 12.5
2021/12/13 23:44:20 Rebalance progress: 12.5
2021/12/13 23:44:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:44:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:44:26 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:44:31 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 23:44:36 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 23:44:41 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 23:44:46 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/13 23:44:46 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/13 23:44:46 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:44:46 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/13 23:44:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:44:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:44:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10614
2021/12/13 23:44:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:44:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:45:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10613
2021/12/13 23:45:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (184.73s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/13 23:45:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 23:45:19 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/13 23:45:19 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:45:19 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/13 23:45:21 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:45:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:45:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:45:33 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:45:33 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:45:34 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/13 23:45:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:45:44 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 23:45:49 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 23:45:54 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/13 23:45:54 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:45:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:45:54 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:45:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:46:00 Rebalance progress: 12.5
2021/12/13 23:46:05 Rebalance progress: 12.5
2021/12/13 23:46:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:46:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:46:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:46:15 Rebalance progress: 12.5
2021/12/13 23:46:20 Rebalance progress: 12.5
2021/12/13 23:46:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:46:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:46:30 Rebalance progress: 12.5
2021/12/13 23:46:35 Rebalance progress: 12.5
2021/12/13 23:46:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:46:40 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:46:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 23:46:51 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 23:46:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 23:47:01 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/13 23:47:01 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/13 23:47:01 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:47:01 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/13 23:47:02 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:47:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:47:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6136
2021/12/13 23:47:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6136
2021/12/13 23:47:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:47:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:47:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:47:29 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6136
2021/12/13 23:47:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.91s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/13 23:47:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:47:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:47:40 Rebalance progress: 100
2021/12/13 23:47:45 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 23:47:45 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/13 23:47:45 No eventing-con process running
2021/12/13 23:47:45 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/13 23:47:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:47:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:47:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:47:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:47:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:47:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:47:58 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:47:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 23:48:04 Starting bucket ops to source bucket
2021/12/13 23:48:09 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 23:48:15 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 23:48:20 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/13 23:48:20 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:48:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:48:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:48:25 Rebalance progress: 12.5
2021/12/13 23:48:30 Rebalance progress: 12.5
2021/12/13 23:48:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:48:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/13 23:48:35 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 23:48:40 Rebalance progress: 12.5
2021/12/13 23:48:45 Rebalance progress: 12.5
2021/12/13 23:48:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:48:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:48:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:48:55 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 23:49:00 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 23:49:05 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 23:49:10 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/13 23:49:10 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/13 23:49:10 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:49:10 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/13 23:49:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:49:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:49:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4558
2021/12/13 23:49:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4558
2021/12/13 23:49:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:49:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:49:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:49:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4558
2021/12/13 23:49:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.75s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/13 23:49:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:49:44 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:49:49 Rebalance progress: 100
2021/12/13 23:49:54 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 23:49:54 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/13 23:49:54 No eventing-con process running
2021/12/13 23:49:54 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/13 23:49:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:50:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:50:03 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:50:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:50:08 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:50:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/13 23:50:14 Starting bucket ops to source bucket
2021/12/13 23:50:19 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 23:50:24 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 23:50:29 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/13 23:50:29 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:50:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:50:30 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:50:35 Rebalance progress: 12.5
2021/12/13 23:50:40 Rebalance progress: 12.5
2021/12/13 23:50:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:50:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:50:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/13 23:50:45 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 23:50:50 Rebalance progress: 12.5
2021/12/13 23:50:55 Rebalance progress: 12.5
2021/12/13 23:51:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:51:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:51:00 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:51:00 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/13 23:51:05 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 23:51:10 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 23:51:15 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 23:51:20 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/13 23:51:20 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/13 23:51:20 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:51:20 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/13 23:51:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:51:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:51:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37808
2021/12/13 23:51:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37808
2021/12/13 23:51:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:51:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:51:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:51:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 37808
2021/12/13 23:51:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (129.68s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/13 23:51:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 23:51:59 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/13 23:51:59 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:51:59 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/13 23:52:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:52:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:52:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:52:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:17 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:52:17 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:52:18 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:52:18 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:52:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 23:52:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:52:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 23:52:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 23:52:40 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/13 23:52:40 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/13 23:52:45 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 23:52:50 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 23:52:55 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/13 23:52:55 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:52:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:52:55 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:52:55 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:53:00 Rebalance progress: 12.5
2021/12/13 23:53:05 Rebalance progress: 12.5
2021/12/13 23:53:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:53:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:53:10 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:53:16 Rebalance progress: 12.5
2021/12/13 23:53:21 Rebalance progress: 12.5
2021/12/13 23:53:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:53:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 23:53:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:53:31 Rebalance progress: 12.5
2021/12/13 23:53:36 Rebalance progress: 12.5
2021/12/13 23:53:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:53:41 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/13 23:53:46 Rebalance progress: 12.5
2021/12/13 23:53:51 Rebalance progress: 12.5
2021/12/13 23:53:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:53:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:53:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/13 23:54:01 Rebalance progress: 12.5
2021/12/13 23:54:06 Rebalance progress: 12.5
2021/12/13 23:54:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:54:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:54:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/13 23:54:16 Rebalance progress: 12.5
2021/12/13 23:54:21 Rebalance progress: 12.5
2021/12/13 23:54:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:54:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:54:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:54:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:54:32 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 23:54:37 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 23:54:42 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 23:54:47 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/13 23:54:47 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/13 23:54:47 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:54:47 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/13 23:54:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:54:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:54:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:54:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:55:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:55:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12190
2021/12/13 23:55:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:55:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:55:12 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 23:55:17 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 23:55:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 23:55:27 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/13 23:55:27 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/13 23:55:27 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:55:27 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/13 23:55:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:55:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:55:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:55:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:55:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:55:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (232.20s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/13 23:55:51 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 23:55:51 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/13 23:55:51 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:55:51 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/13 23:55:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:55:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:55:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:55:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:56:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:56:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:56:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:56:09 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:56:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:56:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:56:11 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:56:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/13 23:56:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:56:22 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 23:56:27 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 23:56:32 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/13 23:56:32 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/13 23:56:37 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 23:56:42 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 23:56:47 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/13 23:56:47 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:56:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:56:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:56:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/13 23:56:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/13 23:56:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/13 23:56:48 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/13 23:56:53 Rebalance progress: 12.5
2021/12/13 23:56:58 Rebalance progress: 12.5
2021/12/13 23:57:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:57:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:57:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:57:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/13 23:57:08 Rebalance progress: 12.5
2021/12/13 23:57:13 Rebalance progress: 12.5
2021/12/13 23:57:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:57:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:57:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/13 23:57:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:57:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 23:57:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 23:57:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 23:57:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/13 23:57:39 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/13 23:57:39 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:57:39 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/13 23:57:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:57:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:57:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4692
2021/12/13 23:57:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4692
2021/12/13 23:57:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:57:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:58:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:58:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6180
2021/12/13 23:58:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:58:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/13 23:58:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 23:58:19 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 23:58:24 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 23:58:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/13 23:58:29 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/13 23:58:29 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:58:29 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/13 23:58:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:58:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:58:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:58:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:58:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:58:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (181.94s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/13 23:58:53 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/13 23:58:53 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/13 23:58:53 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:58:53 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/13 23:58:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:59:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:59:11 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/13 23:59:11 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/13 23:59:11 eventingSetup.go:46: No eventing-con process running
2021/12/13 23:59:11 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/13 23:59:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/13 23:59:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:25 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:59:26 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/13 23:59:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/13 23:59:26 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/13 23:59:27 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/13 23:59:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/13 23:59:37 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 23:59:42 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 23:59:47 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/13 23:59:47 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/13 23:59:52 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/13 23:59:57 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 00:00:02 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/14 00:00:02 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:00:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:00:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:00:08 Rebalance progress: 12.5
2021/12/14 00:00:13 Rebalance progress: 12.5
2021/12/14 00:00:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:00:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 00:00:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:00:23 Rebalance progress: 12.5
2021/12/14 00:00:28 Rebalance progress: 12.5
2021/12/14 00:00:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:00:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:00:39 Rebalance progress: 12.5
2021/12/14 00:00:44 Rebalance progress: 12.5
2021/12/14 00:00:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:00:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:00:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:00:54 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 00:00:59 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 00:01:04 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 00:01:09 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/14 00:01:09 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/14 00:01:09 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:01:09 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/14 00:01:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:01:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:01:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6392
2021/12/14 00:01:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6392
2021/12/14 00:01:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:01:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:01:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:01:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:01:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:01:40 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 00:01:45 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 00:01:50 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 00:01:55 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/14 00:01:55 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/14 00:01:55 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:01:55 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/14 00:01:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:02:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:02:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:02:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (205.53s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 00:02:18 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:02:18 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 00:02:18 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:02:18 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 00:02:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:02:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:02:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:02:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:02:37 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:02:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 00:02:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 00:02:48 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:02:53 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:02:58 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:02:58 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:02:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:02:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:02:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:03:18 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:03:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:03:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:03:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:03:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:03:49 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:03:54 Rebalance progress: 12.5
2021/12/14 00:03:59 Rebalance progress: 12.5
2021/12/14 00:04:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:04:04 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 00:04:05 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:04:10 Rebalance progress: 12.5
2021/12/14 00:04:15 Rebalance progress: 12.5
2021/12/14 00:04:20 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:04:20 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:04:25 Rebalance progress: 12.5
2021/12/14 00:04:30 Rebalance progress: 12.5
2021/12/14 00:04:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:04:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:04:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:04:40 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:04:45 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:04:50 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:04:55 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:04:55 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 00:04:55 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:04:55 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 00:04:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:05:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:05:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11172
2021/12/14 00:05:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:05:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:05:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:05:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11172
2021/12/14 00:05:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (190.29s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 00:05:29 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:05:29 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 00:05:29 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:05:29 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/14 00:05:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:05:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:05:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:05:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:05:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:05:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:05:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:05:47 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:05:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/14 00:05:53 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 00:05:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:06:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:06:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/14 00:06:08 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:06:08 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:06:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:06:29 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:06:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:06:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:06:40 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:07:00 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:07:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:07:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:07:31 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:07:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:07:41 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:07:41 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:08:02 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:08:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:08:12 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:08:32 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:08:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:08:48 Rebalance progress: 12.5
2021/12/14 00:08:53 Rebalance progress: 12.5
2021/12/14 00:08:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:08:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:08:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 00:08:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:09:03 Rebalance progress: 12.5
2021/12/14 00:09:08 Rebalance progress: 12.5
2021/12/14 00:09:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:09:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:09:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:09:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:09:19 Rebalance progress: 12.5
2021/12/14 00:09:24 Rebalance progress: 12.5
2021/12/14 00:09:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:09:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:09:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:09:34 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:09:39 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:09:44 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:09:49 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/14 00:09:49 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/14 00:09:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:09:49 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/14 00:09:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:09:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:10:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13384
2021/12/14 00:10:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 13384
2021/12/14 00:10:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:10:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:10:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:10:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21523
2021/12/14 00:10:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (298.91s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/14 00:10:27 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 00:10:27 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 00:10:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:10:27 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/14 00:10:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:10:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:10:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:10:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:10:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:10:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:10:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:10:46 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:10:47 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 00:10:52 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 00:10:57 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 00:11:02 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 00:11:07 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/14 00:11:07 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:11:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:11:08 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:11:13 Rebalance progress: 12.5
2021/12/14 00:11:18 Rebalance progress: 12.5
2021/12/14 00:11:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:11:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:11:33 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:11:38 Rebalance progress: 12.5
2021/12/14 00:11:43 Rebalance progress: 12.5
2021/12/14 00:11:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:11:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:11:48 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 00:11:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:11:53 Rebalance progress: 12.5
2021/12/14 00:11:58 Rebalance progress: 12.5
2021/12/14 00:12:03 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:12:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:12:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:12:03 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:12:08 Rebalance progress: 12.5
2021/12/14 00:12:13 Rebalance progress: 12.5
2021/12/14 00:12:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:12:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:12:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:12:23 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 00:12:28 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 00:12:33 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 00:12:38 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/14 00:12:38 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/14 00:12:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:12:38 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/14 00:12:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:12:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:12:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2641
2021/12/14 00:12:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2641
2021/12/14 00:12:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:12:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:13:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8607
2021/12/14 00:13:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.59s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/14 00:13:17 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 00:13:17 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 00:13:17 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:13:17 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/14 00:13:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:13:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:13:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:13:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:13:36 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:13:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/14 00:13:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 00:13:47 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 00:13:52 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 00:13:57 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/14 00:13:57 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:13:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:13:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/14 00:13:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:14:02 Rebalance progress: 12.5
2021/12/14 00:14:07 Rebalance progress: 12.5
2021/12/14 00:14:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:14:12 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/14 00:14:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:14:18 Rebalance progress: 12.5
2021/12/14 00:14:23 Rebalance progress: 12.5
2021/12/14 00:14:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:14:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:14:33 Rebalance progress: 12.5
2021/12/14 00:14:38 Rebalance progress: 12.5
2021/12/14 00:14:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:14:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:14:43 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:14:48 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 00:14:53 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 00:14:58 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 00:15:03 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/14 00:15:03 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/14 00:15:03 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:15:03 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/14 00:15:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:15:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:15:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6087
2021/12/14 00:15:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:15:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:15:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:15:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6087
2021/12/14 00:15:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.58s)
=== RUN   TestEventingRebInWithOps
2021/12/14 00:15:37 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 00:15:37 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/14 00:15:37 eventingSetup.go:46: No eventing-con process running
2021/12/14 00:15:37 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/14 00:15:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:15:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:15:43 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:15:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:15:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:15:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:15:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:15:55 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:15:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:16:01 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 00:16:06 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 00:16:11 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 00:16:16 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/14 00:16:16 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/14 00:16:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:16:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 00:16:17 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 00:16:22 Rebalance progress: 12.5
2021/12/14 00:16:27 Rebalance progress: 12.5
2021/12/14 00:16:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:16:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:16:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:16:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:16:37 Rebalance progress: 12.5
2021/12/14 00:16:42 Rebalance progress: 12.5
2021/12/14 00:16:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:16:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:16:47 Waiting for src bucket item count to get to: 200000 curr count: 34108
2021/12/14 00:16:52 Waiting for src bucket item count to get to: 200000 curr count: 49553
2021/12/14 00:16:57 Waiting for src bucket item count to get to: 200000 curr count: 49553
2021/12/14 00:17:02 Waiting for src bucket item count to get to: 200000 curr count: 65435
2021/12/14 00:17:07 Waiting for src bucket item count to get to: 200000 curr count: 65435
2021/12/14 00:17:12 Waiting for src bucket item count to get to: 200000 curr count: 79882
2021/12/14 00:17:17 Waiting for src bucket item count to get to: 200000 curr count: 79882
2021/12/14 00:17:22 Waiting for src bucket item count to get to: 200000 curr count: 95757
2021/12/14 00:17:27 Waiting for src bucket item count to get to: 200000 curr count: 95757
2021/12/14 00:17:32 Waiting for src bucket item count to get to: 200000 curr count: 111709
2021/12/14 00:17:37 Waiting for src bucket item count to get to: 200000 curr count: 111709
2021/12/14 00:17:42 Waiting for src bucket item count to get to: 200000 curr count: 111709
2021/12/14 00:17:47 Waiting for src bucket item count to get to: 200000 curr count: 127707
2021/12/14 00:17:52 Waiting for src bucket item count to get to: 200000 curr count: 127707
2021/12/14 00:17:57 Waiting for src bucket item count to get to: 200000 curr count: 143513
2021/12/14 00:18:02 Waiting for src bucket item count to get to: 200000 curr count: 143513
2021/12/14 00:18:07 Waiting for src bucket item count to get to: 200000 curr count: 159407
2021/12/14 00:18:12 Waiting for src bucket item count to get to: 200000 curr count: 159407
2021/12/14 00:18:17 Waiting for src bucket item count to get to: 200000 curr count: 173803
2021/12/14 00:18:22 Waiting for src bucket item count to get to: 200000 curr count: 173803
2021/12/14 00:18:27 Waiting for src bucket item count to get to: 200000 curr count: 189930
2021/12/14 00:18:32 Waiting for src bucket item count to get to: 200000 curr count: 189930
2021/12/14 00:18:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:18:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:18:37 Starting bucket ops to source bucket
2021/12/14 00:22:28 Waiting for dst bucket item count to get to: 0 curr count: 6159
2021/12/14 00:22:33 Waiting for dst bucket item count to get to: 0 curr count: 6159
2021/12/14 00:22:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:22:33 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 00:22:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:22:38 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 00:22:43 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 00:22:48 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 00:22:53 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/14 00:22:53 App: TestEventingRebInWithOps got un-deployed
2021/12/14 00:22:53 No eventing-con process running
2021/12/14 00:22:53 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/14 00:22:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:23:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:23:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:23:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:23:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:23:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (459.87s)
=== RUN   TestEventingRebOutWithOps
2021/12/14 00:23:11 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:23:12 Removing node(s): [] from the cluster
2021/12/14 00:23:17 Rebalance progress: 100
2021/12/14 00:23:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 00:23:22 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 00:23:22 No eventing-con process running
2021/12/14 00:23:22 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/14 00:23:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:23:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:23:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:23:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:23:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:23:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:23:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:23:40 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:23:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:23:46 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 00:23:51 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 00:23:56 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 00:24:01 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/14 00:24:01 App: TestEventingRebOutWithOps got deployed
2021/12/14 00:24:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:24:01 Starting bucket ops to source bucket
2021/12/14 00:24:07 Rebalance progress: 12.5
2021/12/14 00:24:12 Rebalance progress: 12.5
2021/12/14 00:24:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:24:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:24:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:24:17 Waiting for src bucket item count to get to: 200000 curr count: 11436
2021/12/14 00:24:22 Waiting for src bucket item count to get to: 200000 curr count: 27392
2021/12/14 00:24:27 Waiting for src bucket item count to get to: 200000 curr count: 27392
2021/12/14 00:24:32 Waiting for src bucket item count to get to: 200000 curr count: 43585
2021/12/14 00:24:37 Waiting for src bucket item count to get to: 200000 curr count: 43585
2021/12/14 00:24:42 Waiting for src bucket item count to get to: 200000 curr count: 59507
2021/12/14 00:24:47 Waiting for src bucket item count to get to: 200000 curr count: 59507
2021/12/14 00:24:52 Waiting for src bucket item count to get to: 200000 curr count: 59507
2021/12/14 00:24:57 Waiting for src bucket item count to get to: 200000 curr count: 74461
2021/12/14 00:25:02 Waiting for src bucket item count to get to: 200000 curr count: 74461
2021/12/14 00:25:07 Waiting for src bucket item count to get to: 200000 curr count: 90375
2021/12/14 00:25:12 Waiting for src bucket item count to get to: 200000 curr count: 90375
2021/12/14 00:25:17 Waiting for src bucket item count to get to: 200000 curr count: 106556
2021/12/14 00:25:22 Waiting for src bucket item count to get to: 200000 curr count: 106556
2021/12/14 00:25:27 Waiting for src bucket item count to get to: 200000 curr count: 122450
2021/12/14 00:25:32 Waiting for src bucket item count to get to: 200000 curr count: 122450
2021/12/14 00:25:37 Waiting for src bucket item count to get to: 200000 curr count: 138295
2021/12/14 00:25:42 Waiting for src bucket item count to get to: 200000 curr count: 138295
2021/12/14 00:25:47 Waiting for src bucket item count to get to: 200000 curr count: 154242
2021/12/14 00:25:52 Waiting for src bucket item count to get to: 200000 curr count: 154242
2021/12/14 00:25:57 Waiting for src bucket item count to get to: 200000 curr count: 169521
2021/12/14 00:26:02 Waiting for src bucket item count to get to: 200000 curr count: 169521
2021/12/14 00:26:07 Waiting for src bucket item count to get to: 200000 curr count: 185349
2021/12/14 00:26:12 Waiting for src bucket item count to get to: 200000 curr count: 185349
2021/12/14 00:26:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:26:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:26:17 Starting bucket ops to source bucket
2021/12/14 00:30:07 Waiting for dst bucket item count to get to: 0 curr count: 5213
2021/12/14 00:30:12 Waiting for dst bucket item count to get to: 0 curr count: 5213
2021/12/14 00:30:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:30:12 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 00:30:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:30:17 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 00:30:22 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 00:30:27 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 00:30:32 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/14 00:30:32 App: TestEventingRebOutWithOps got un-deployed
2021/12/14 00:30:32 No eventing-con process running
2021/12/14 00:30:32 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/14 00:30:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:30:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:30:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:30:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:30:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:30:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (459.55s)
=== RUN   TestEventingSwapRebWithOps
2021/12/14 00:30:56 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 00:30:56 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 00:30:56 No eventing-con process running
2021/12/14 00:30:56 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/14 00:30:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:31:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:31:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:31:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:31:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:31:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:31:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:31:15 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:31:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:31:21 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 00:31:26 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 00:31:31 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 00:31:36 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/14 00:31:36 App: TestEventingSwapRebWithOps got deployed
2021/12/14 00:31:36 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 00:31:36 Starting bucket ops to source bucket
2021/12/14 00:31:36 Removing node(s): [] from the cluster
2021/12/14 00:31:41 Rebalance progress: 12.5
2021/12/14 00:31:46 Rebalance progress: 12.5
2021/12/14 00:31:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:31:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:31:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:31:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 00:31:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:31:57 Rebalance progress: 12.5
2021/12/14 00:32:02 Rebalance progress: 12.5
2021/12/14 00:32:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:32:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:32:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:32:12 Rebalance progress: 12.5
2021/12/14 00:32:17 Rebalance progress: 12.5
2021/12/14 00:32:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:32:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:32:22 Waiting for src bucket item count to get to: 200000 curr count: 66371
2021/12/14 00:32:27 Waiting for src bucket item count to get to: 200000 curr count: 66371
2021/12/14 00:32:32 Waiting for src bucket item count to get to: 200000 curr count: 80865
2021/12/14 00:32:37 Waiting for src bucket item count to get to: 200000 curr count: 80865
2021/12/14 00:32:42 Waiting for src bucket item count to get to: 200000 curr count: 97031
2021/12/14 00:32:47 Waiting for src bucket item count to get to: 200000 curr count: 97031
2021/12/14 00:32:52 Waiting for src bucket item count to get to: 200000 curr count: 113249
2021/12/14 00:32:57 Waiting for src bucket item count to get to: 200000 curr count: 113249
2021/12/14 00:33:02 Waiting for src bucket item count to get to: 200000 curr count: 129607
2021/12/14 00:33:07 Waiting for src bucket item count to get to: 200000 curr count: 129607
2021/12/14 00:33:12 Waiting for src bucket item count to get to: 200000 curr count: 145595
2021/12/14 00:33:17 Waiting for src bucket item count to get to: 200000 curr count: 145595
2021/12/14 00:33:22 Waiting for src bucket item count to get to: 200000 curr count: 161699
2021/12/14 00:33:27 Waiting for src bucket item count to get to: 200000 curr count: 161699
2021/12/14 00:33:32 Waiting for src bucket item count to get to: 200000 curr count: 176707
2021/12/14 00:33:37 Waiting for src bucket item count to get to: 200000 curr count: 176707
2021/12/14 00:33:42 Waiting for src bucket item count to get to: 200000 curr count: 192989
2021/12/14 00:33:47 Waiting for src bucket item count to get to: 200000 curr count: 192989
2021/12/14 00:33:52 Waiting for src bucket item count to get to: 200000 curr count: 192989
2021/12/14 00:33:57 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:33:57 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:33:57 Starting bucket ops to source bucket
2021/12/14 00:37:48 Waiting for dst bucket item count to get to: 0 curr count: 6204
2021/12/14 00:37:53 Waiting for dst bucket item count to get to: 0 curr count: 6204
2021/12/14 00:37:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:37:53 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 00:37:53 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:37:58 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 00:38:03 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 00:38:08 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 00:38:13 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/14 00:38:13 App: TestEventingSwapRebWithOps got un-deployed
2021/12/14 00:38:13 No eventing-con process running
2021/12/14 00:38:13 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/14 00:38:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:38:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:38:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:38:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:38:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (460.49s)
=== RUN   TestKVRebInWithOps
2021/12/14 00:38:37 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 00:38:37 App: TestKVRebInWithOps got un-deployed
2021/12/14 00:38:37 No eventing-con process running
2021/12/14 00:38:37 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/14 00:38:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:38:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:38:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:38:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:38:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:38:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:38:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:38:55 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:38:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:39:01 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 00:39:06 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 00:39:11 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 00:39:16 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/14 00:39:16 App: TestKVRebInWithOps got deployed
2021/12/14 00:39:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 00:39:16 Starting bucket ops to source bucket
2021/12/14 00:39:16 Removing node(s): [] from the cluster
2021/12/14 00:39:22 Rebalance progress: 12.5
2021/12/14 00:39:27 Rebalance progress: 12.5
2021/12/14 00:39:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:39:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:39:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:39:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:39:37 Rebalance progress: 12.5
2021/12/14 00:39:42 Rebalance progress: 12.5
2021/12/14 00:39:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:39:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:39:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:39:47 Waiting for src bucket item count to get to: 200000 curr count: 34875
2021/12/14 00:39:52 Waiting for src bucket item count to get to: 200000 curr count: 50790
2021/12/14 00:39:57 Waiting for src bucket item count to get to: 200000 curr count: 50790
2021/12/14 00:40:02 Waiting for src bucket item count to get to: 200000 curr count: 66843
2021/12/14 00:40:07 Waiting for src bucket item count to get to: 200000 curr count: 66843
2021/12/14 00:40:12 Waiting for src bucket item count to get to: 200000 curr count: 81736
2021/12/14 00:40:17 Waiting for src bucket item count to get to: 200000 curr count: 81736
2021/12/14 00:40:22 Waiting for src bucket item count to get to: 200000 curr count: 97795
2021/12/14 00:40:27 Waiting for src bucket item count to get to: 200000 curr count: 97795
2021/12/14 00:40:32 Waiting for src bucket item count to get to: 200000 curr count: 97795
2021/12/14 00:40:37 Waiting for src bucket item count to get to: 200000 curr count: 113932
2021/12/14 00:40:42 Waiting for src bucket item count to get to: 200000 curr count: 113932
2021/12/14 00:40:47 Waiting for src bucket item count to get to: 200000 curr count: 129807
2021/12/14 00:40:52 Waiting for src bucket item count to get to: 200000 curr count: 129807
2021/12/14 00:40:57 Waiting for src bucket item count to get to: 200000 curr count: 145821
2021/12/14 00:41:02 Waiting for src bucket item count to get to: 200000 curr count: 145821
2021/12/14 00:41:07 Waiting for src bucket item count to get to: 200000 curr count: 161666
2021/12/14 00:41:12 Waiting for src bucket item count to get to: 200000 curr count: 161666
2021/12/14 00:41:17 Waiting for src bucket item count to get to: 200000 curr count: 176264
2021/12/14 00:41:22 Waiting for src bucket item count to get to: 200000 curr count: 176264
2021/12/14 00:41:27 Waiting for src bucket item count to get to: 200000 curr count: 192574
2021/12/14 00:41:32 Waiting for src bucket item count to get to: 200000 curr count: 192574
2021/12/14 00:41:37 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:41:37 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:41:37 Starting bucket ops to source bucket
2021/12/14 00:45:27 Waiting for dst bucket item count to get to: 0 curr count: 4973
2021/12/14 00:45:32 Waiting for dst bucket item count to get to: 0 curr count: 4973
2021/12/14 00:45:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:45:32 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 00:45:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:45:37 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 00:45:42 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 00:45:47 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 00:45:52 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/14 00:45:52 App: TestKVRebInWithOps got un-deployed
2021/12/14 00:45:52 No eventing-con process running
2021/12/14 00:45:52 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/14 00:45:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:45:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:45:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:46:01 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:46:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:46:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (459.27s)
=== RUN   TestKVRebOutWithOps
2021/12/14 00:46:11 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 00:46:11 Removing node(s): [] from the cluster
2021/12/14 00:46:16 Rebalance progress: 100
2021/12/14 00:46:21 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 00:46:21 App: TestKVRebOutWithOps got un-deployed
2021/12/14 00:46:21 No eventing-con process running
2021/12/14 00:46:21 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/14 00:46:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:46:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:46:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:46:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:46:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:46:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:46:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:46:40 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:46:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:46:46 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 00:46:51 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 00:46:56 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 00:47:01 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/14 00:47:01 App: TestKVRebOutWithOps got deployed
2021/12/14 00:47:01 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:47:01 Starting bucket ops to source bucket
2021/12/14 00:47:06 Rebalance progress: 12.5
2021/12/14 00:47:11 Rebalance progress: 12.5
2021/12/14 00:47:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:47:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:47:16 Waiting for src bucket item count to get to: 200000 curr count: 12477
2021/12/14 00:47:21 Waiting for src bucket item count to get to: 200000 curr count: 28401
2021/12/14 00:47:26 Waiting for src bucket item count to get to: 200000 curr count: 28401
2021/12/14 00:47:31 Waiting for src bucket item count to get to: 200000 curr count: 44422
2021/12/14 00:47:36 Waiting for src bucket item count to get to: 200000 curr count: 44422
2021/12/14 00:47:41 Waiting for src bucket item count to get to: 200000 curr count: 60428
2021/12/14 00:47:46 Waiting for src bucket item count to get to: 200000 curr count: 60428
2021/12/14 00:47:51 Waiting for src bucket item count to get to: 200000 curr count: 60428
2021/12/14 00:47:56 Waiting for src bucket item count to get to: 200000 curr count: 75433
2021/12/14 00:48:01 Waiting for src bucket item count to get to: 200000 curr count: 75433
2021/12/14 00:48:06 Waiting for src bucket item count to get to: 200000 curr count: 91107
2021/12/14 00:48:11 Waiting for src bucket item count to get to: 200000 curr count: 91107
2021/12/14 00:48:16 Waiting for src bucket item count to get to: 200000 curr count: 107136
2021/12/14 00:48:21 Waiting for src bucket item count to get to: 200000 curr count: 107136
2021/12/14 00:48:26 Waiting for src bucket item count to get to: 200000 curr count: 123198
2021/12/14 00:48:31 Waiting for src bucket item count to get to: 200000 curr count: 123198
2021/12/14 00:48:36 Waiting for src bucket item count to get to: 200000 curr count: 139157
2021/12/14 00:48:41 Waiting for src bucket item count to get to: 200000 curr count: 139157
2021/12/14 00:48:46 Waiting for src bucket item count to get to: 200000 curr count: 155220
2021/12/14 00:48:51 Waiting for src bucket item count to get to: 200000 curr count: 155220
2021/12/14 00:48:56 Waiting for src bucket item count to get to: 200000 curr count: 170355
2021/12/14 00:49:01 Waiting for src bucket item count to get to: 200000 curr count: 170355
2021/12/14 00:49:06 Waiting for src bucket item count to get to: 200000 curr count: 186047
2021/12/14 00:49:11 Waiting for src bucket item count to get to: 200000 curr count: 186047
2021/12/14 00:49:16 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:49:16 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:49:16 Starting bucket ops to source bucket
2021/12/14 00:53:07 Waiting for dst bucket item count to get to: 0 curr count: 4326
2021/12/14 00:53:12 Waiting for dst bucket item count to get to: 0 curr count: 4326
2021/12/14 00:53:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:53:12 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 00:53:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 00:53:17 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 00:53:22 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 00:53:27 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 00:53:32 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/14 00:53:32 App: TestKVRebOutWithOps got un-deployed
2021/12/14 00:53:32 No eventing-con process running
2021/12/14 00:53:32 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/14 00:53:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:53:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:53:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:53:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:53:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (459.74s)
=== RUN   TestKVSwapRebWithOps
2021/12/14 00:53:56 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 00:53:56 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 00:53:56 No eventing-con process running
2021/12/14 00:53:56 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/14 00:53:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:54:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:54:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 00:54:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:54:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:54:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:54:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 00:54:14 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/14 00:54:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/14 00:54:20 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 00:54:25 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 00:54:30 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 00:54:35 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/14 00:54:35 App: TestKVSwapRebWithOps got deployed
2021/12/14 00:54:35 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 00:54:35 Starting bucket ops to source bucket
2021/12/14 00:54:36 Removing node(s): [] from the cluster
2021/12/14 00:54:41 Rebalance progress: 12.5
2021/12/14 00:54:46 Rebalance progress: 12.5
2021/12/14 00:54:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:54:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:54:51 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 00:54:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 00:54:56 Rebalance progress: 12.5
2021/12/14 00:55:01 Rebalance progress: 12.5
2021/12/14 00:55:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:55:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:55:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 00:55:11 Rebalance progress: 12.5
2021/12/14 00:55:16 Rebalance progress: 12.5
2021/12/14 00:55:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:55:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 00:55:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 00:55:21 Waiting for src bucket item count to get to: 200000 curr count: 67208
2021/12/14 00:55:26 Waiting for src bucket item count to get to: 200000 curr count: 67208
2021/12/14 00:55:31 Waiting for src bucket item count to get to: 200000 curr count: 81444
2021/12/14 00:55:36 Waiting for src bucket item count to get to: 200000 curr count: 81444
2021/12/14 00:55:41 Waiting for src bucket item count to get to: 200000 curr count: 97400
2021/12/14 00:55:46 Waiting for src bucket item count to get to: 200000 curr count: 97400
2021/12/14 00:55:51 Waiting for src bucket item count to get to: 200000 curr count: 113384
2021/12/14 00:55:56 Waiting for src bucket item count to get to: 200000 curr count: 113384
2021/12/14 00:56:01 Waiting for src bucket item count to get to: 200000 curr count: 129458
2021/12/14 00:56:06 Waiting for src bucket item count to get to: 200000 curr count: 129458
2021/12/14 00:56:11 Waiting for src bucket item count to get to: 200000 curr count: 145772
2021/12/14 00:56:16 Waiting for src bucket item count to get to: 200000 curr count: 145772
2021/12/14 00:56:21 Waiting for src bucket item count to get to: 200000 curr count: 145772
2021/12/14 00:56:26 Waiting for src bucket item count to get to: 200000 curr count: 161704
2021/12/14 00:56:31 Waiting for src bucket item count to get to: 200000 curr count: 161704
2021/12/14 00:56:36 Waiting for src bucket item count to get to: 200000 curr count: 176658
2021/12/14 00:56:41 Waiting for src bucket item count to get to: 200000 curr count: 176658
2021/12/14 00:56:46 Waiting for src bucket item count to get to: 200000 curr count: 192747
2021/12/14 00:56:51 Waiting for src bucket item count to get to: 200000 curr count: 192747
2021/12/14 00:56:56 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/14 00:56:56 Post SET ops. Expected item count: 200000 got 200000
2021/12/14 00:56:56 Starting bucket ops to source bucket
2021/12/14 01:00:47 Waiting for dst bucket item count to get to: 0 curr count: 4471
2021/12/14 01:00:52 Waiting for dst bucket item count to get to: 0 curr count: 4471
2021/12/14 01:00:52 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:00:52 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:00:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:00:57 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 01:01:02 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 01:01:07 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 01:01:12 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/14 01:01:12 App: TestKVSwapRebWithOps got un-deployed
2021/12/14 01:01:12 No eventing-con process running
2021/12/14 01:01:12 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/14 01:01:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:01:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:01:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:01:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:01:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (459.97s)
=== RUN   TestEventingRebInWithTimers
2021/12/14 01:01:35 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 01:01:35 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 01:01:36 No eventing-con process running
2021/12/14 01:01:36 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/14 01:01:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:01:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:01:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:01:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:01:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:01:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:01:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:01:54 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:01:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:02:00 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 01:02:05 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 01:02:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 01:02:15 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/14 01:02:15 App: TestEventingRebInWithTimers got deployed
2021/12/14 01:02:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:02:15 Starting bucket ops to source bucket
2021/12/14 01:02:16 Removing node(s): [] from the cluster
2021/12/14 01:02:21 Rebalance progress: 12.5
2021/12/14 01:02:26 Rebalance progress: 12.5
2021/12/14 01:02:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:02:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:02:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:02:37 Rebalance progress: 12.5
2021/12/14 01:02:42 Rebalance progress: 12.5
2021/12/14 01:02:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:02:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:02:47 Waiting for src bucket item count to get to: 50000 curr count: 28345
2021/12/14 01:02:52 Waiting for src bucket item count to get to: 50000 curr count: 40061
2021/12/14 01:02:57 Waiting for src bucket item count to get to: 50000 curr count: 40061
2021/12/14 01:03:07 Waiting for dst bucket item count to get to: 50000 curr count: 38827
2021/12/14 01:03:12 Waiting for dst bucket item count to get to: 50000 curr count: 38827
2021/12/14 01:03:17 Waiting for dst bucket item count to get to: 50000 curr count: 46951
2021/12/14 01:03:22 Waiting for dst bucket item count to get to: 50000 curr count: 46951
2021/12/14 01:03:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:03:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:03:22 Starting bucket ops to source bucket
2021/12/14 01:04:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:04:37 Waiting for dst bucket item count to get to: 0 curr count: 49974
2021/12/14 01:04:42 Waiting for dst bucket item count to get to: 0 curr count: 49974
2021/12/14 01:04:47 Waiting for dst bucket item count to get to: 0 curr count: 43217
2021/12/14 01:04:52 Waiting for dst bucket item count to get to: 0 curr count: 43217
2021/12/14 01:04:57 Waiting for dst bucket item count to get to: 0 curr count: 43217
2021/12/14 01:05:02 Waiting for dst bucket item count to get to: 0 curr count: 25148
2021/12/14 01:05:07 Waiting for dst bucket item count to get to: 0 curr count: 25148
2021/12/14 01:05:12 Waiting for dst bucket item count to get to: 0 curr count: 4234
2021/12/14 01:05:17 Waiting for dst bucket item count to get to: 0 curr count: 4234
2021/12/14 01:05:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:05:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:05:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:05:22 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 01:05:27 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 01:05:32 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 01:05:37 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/14 01:05:37 App: TestEventingRebInWithTimers got un-deployed
2021/12/14 01:05:37 No eventing-con process running
2021/12/14 01:05:37 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/14 01:05:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:05:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:05:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:05:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:05:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:05:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (265.38s)
=== RUN   TestEventingRebOutWithTimers
2021/12/14 01:05:56 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:05:56 Removing node(s): [] from the cluster
2021/12/14 01:06:01 Rebalance progress: 100
2021/12/14 01:06:06 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 01:06:06 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 01:06:06 No eventing-con process running
2021/12/14 01:06:06 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/14 01:06:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:06:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:06:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:06:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:06:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:06:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:06:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:06:25 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:06:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:06:31 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 01:06:36 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 01:06:41 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 01:06:46 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/14 01:06:46 App: TestEventingRebOutWithTimers got deployed
2021/12/14 01:06:46 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:06:46 Starting bucket ops to source bucket
2021/12/14 01:06:51 Rebalance progress: 12.5
2021/12/14 01:06:56 Rebalance progress: 12.5
2021/12/14 01:07:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:07:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:07:01 Waiting for src bucket item count to get to: 50000 curr count: 18569
2021/12/14 01:07:06 Waiting for src bucket item count to get to: 50000 curr count: 18569
2021/12/14 01:07:11 Waiting for src bucket item count to get to: 50000 curr count: 30163
2021/12/14 01:07:16 Waiting for src bucket item count to get to: 50000 curr count: 30163
2021/12/14 01:07:21 Waiting for src bucket item count to get to: 50000 curr count: 41619
2021/12/14 01:07:26 Waiting for src bucket item count to get to: 50000 curr count: 41619
2021/12/14 01:07:36 Waiting for dst bucket item count to get to: 50000 curr count: 35483
2021/12/14 01:07:41 Waiting for dst bucket item count to get to: 50000 curr count: 35483
2021/12/14 01:07:46 Waiting for dst bucket item count to get to: 50000 curr count: 35483
2021/12/14 01:07:51 Waiting for dst bucket item count to get to: 50000 curr count: 42765
2021/12/14 01:07:56 Waiting for dst bucket item count to get to: 50000 curr count: 42765
2021/12/14 01:07:56 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:07:56 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:07:56 Starting bucket ops to source bucket
2021/12/14 01:09:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:09:11 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:09:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:09:21 Waiting for dst bucket item count to get to: 0 curr count: 48627
2021/12/14 01:09:26 Waiting for dst bucket item count to get to: 0 curr count: 48627
2021/12/14 01:09:31 Waiting for dst bucket item count to get to: 0 curr count: 38293
2021/12/14 01:09:36 Waiting for dst bucket item count to get to: 0 curr count: 38293
2021/12/14 01:09:41 Waiting for dst bucket item count to get to: 0 curr count: 18391
2021/12/14 01:09:46 Waiting for dst bucket item count to get to: 0 curr count: 18391
2021/12/14 01:09:51 Waiting for dst bucket item count to get to: 0 curr count: 6335
2021/12/14 01:09:56 Waiting for dst bucket item count to get to: 0 curr count: 6335
2021/12/14 01:09:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:09:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:09:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:10:01 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 01:10:06 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 01:10:11 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 01:10:16 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/14 01:10:16 App: TestEventingRebOutWithTimers got un-deployed
2021/12/14 01:10:16 No eventing-con process running
2021/12/14 01:10:16 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/14 01:10:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:10:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:10:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:10:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:10:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:10:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (279.20s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/14 01:10:40 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 01:10:40 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 01:10:40 No eventing-con process running
2021/12/14 01:10:40 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/14 01:10:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:10:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:10:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:10:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:10:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:10:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:10:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:10:59 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:11:00 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:11:05 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 01:11:10 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 01:11:15 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 01:11:20 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/14 01:11:20 App: TestEventingSwapRebWithTimers got deployed
2021/12/14 01:11:20 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:11:20 Starting bucket ops to source bucket
2021/12/14 01:11:21 Removing node(s): [] from the cluster
2021/12/14 01:11:26 Rebalance progress: 12.5
2021/12/14 01:11:31 Rebalance progress: 12.5
2021/12/14 01:11:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:11:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:11:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 01:11:36 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:11:41 Rebalance progress: 12.5
2021/12/14 01:11:46 Rebalance progress: 12.5
2021/12/14 01:11:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:11:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:11:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:11:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 01:11:56 Rebalance progress: 12.5
2021/12/14 01:12:01 Rebalance progress: 12.5
2021/12/14 01:12:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:12:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:12:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:12:06 Waiting for src bucket item count to get to: 50000 curr count: 45580
2021/12/14 01:12:16 Waiting for dst bucket item count to get to: 50000 curr count: 44836
2021/12/14 01:12:21 Waiting for dst bucket item count to get to: 50000 curr count: 44836
2021/12/14 01:12:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:12:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:12:21 Starting bucket ops to source bucket
2021/12/14 01:13:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:13:37 Waiting for dst bucket item count to get to: 0 curr count: 49137
2021/12/14 01:13:42 Waiting for dst bucket item count to get to: 0 curr count: 49137
2021/12/14 01:13:47 Waiting for dst bucket item count to get to: 0 curr count: 38398
2021/12/14 01:13:52 Waiting for dst bucket item count to get to: 0 curr count: 38398
2021/12/14 01:13:57 Waiting for dst bucket item count to get to: 0 curr count: 17217
2021/12/14 01:14:02 Waiting for dst bucket item count to get to: 0 curr count: 17217
2021/12/14 01:14:07 Waiting for dst bucket item count to get to: 0 curr count: 17217
2021/12/14 01:14:12 Waiting for dst bucket item count to get to: 0 curr count: 5461
2021/12/14 01:14:17 Waiting for dst bucket item count to get to: 0 curr count: 5461
2021/12/14 01:14:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:14:17 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:14:17 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:14:22 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 01:14:27 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 01:14:32 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 01:14:37 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/14 01:14:37 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/14 01:14:37 No eventing-con process running
2021/12/14 01:14:37 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/14 01:14:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:14:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:14:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:14:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:14:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:14:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (260.61s)
=== RUN   TestKVRebInWithTimers
2021/12/14 01:15:01 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 01:15:01 App: TestKVRebInWithTimers got un-deployed
2021/12/14 01:15:01 No eventing-con process running
2021/12/14 01:15:01 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/14 01:15:02 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:15:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:15:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:15:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:15:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:15:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:15:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:15:19 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:15:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:15:25 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 01:15:30 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 01:15:35 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 01:15:40 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/14 01:15:40 App: TestKVRebInWithTimers got deployed
2021/12/14 01:15:40 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:15:40 Starting bucket ops to source bucket
2021/12/14 01:15:41 Removing node(s): [] from the cluster
2021/12/14 01:15:46 Rebalance progress: 12.5
2021/12/14 01:15:51 Rebalance progress: 12.5
2021/12/14 01:15:56 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:15:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:15:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:15:56 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:16:01 Rebalance progress: 12.5
2021/12/14 01:16:06 Rebalance progress: 12.5
2021/12/14 01:16:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:16:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:16:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:16:11 Waiting for src bucket item count to get to: 50000 curr count: 33577
2021/12/14 01:16:16 Waiting for src bucket item count to get to: 50000 curr count: 33577
2021/12/14 01:16:21 Waiting for src bucket item count to get to: 50000 curr count: 44895
2021/12/14 01:16:26 Waiting for src bucket item count to get to: 50000 curr count: 44895
2021/12/14 01:16:36 Waiting for dst bucket item count to get to: 50000 curr count: 40999
2021/12/14 01:16:41 Waiting for dst bucket item count to get to: 50000 curr count: 40999
2021/12/14 01:16:41 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:16:41 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:16:41 Starting bucket ops to source bucket
2021/12/14 01:17:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:17:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:18:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:18:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:18:12 Waiting for dst bucket item count to get to: 0 curr count: 43516
2021/12/14 01:18:17 Waiting for dst bucket item count to get to: 0 curr count: 43516
2021/12/14 01:18:22 Waiting for dst bucket item count to get to: 0 curr count: 22824
2021/12/14 01:18:27 Waiting for dst bucket item count to get to: 0 curr count: 22824
2021/12/14 01:18:32 Waiting for dst bucket item count to get to: 0 curr count: 7246
2021/12/14 01:18:37 Waiting for dst bucket item count to get to: 0 curr count: 7246
2021/12/14 01:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:18:37 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:18:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:18:42 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 01:18:47 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 01:18:52 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 01:18:57 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/14 01:18:57 App: TestKVRebInWithTimers got un-deployed
2021/12/14 01:18:57 No eventing-con process running
2021/12/14 01:18:57 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/14 01:18:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:19:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:19:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:19:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:19:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:19:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (260.04s)
=== RUN   TestKVRebOutWithTimers
2021/12/14 01:19:16 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:19:16 Removing node(s): [] from the cluster
2021/12/14 01:19:21 Rebalance progress: 100
2021/12/14 01:19:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 01:19:26 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 01:19:26 No eventing-con process running
2021/12/14 01:19:26 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/14 01:19:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:19:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:19:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:19:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:19:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:19:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:19:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:19:45 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:19:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:19:51 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 01:19:56 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 01:20:01 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 01:20:06 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/14 01:20:06 App: TestKVRebOutWithTimers got deployed
2021/12/14 01:20:06 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:20:06 Starting bucket ops to source bucket
2021/12/14 01:20:11 Rebalance progress: 12.5
2021/12/14 01:20:16 Rebalance progress: 12.5
2021/12/14 01:20:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:20:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:20:21 Waiting for src bucket item count to get to: 50000 curr count: 18724
2021/12/14 01:20:26 Waiting for src bucket item count to get to: 50000 curr count: 18724
2021/12/14 01:20:31 Waiting for src bucket item count to get to: 50000 curr count: 28930
2021/12/14 01:20:36 Waiting for src bucket item count to get to: 50000 curr count: 28930
2021/12/14 01:20:41 Waiting for src bucket item count to get to: 50000 curr count: 40804
2021/12/14 01:20:46 Waiting for src bucket item count to get to: 50000 curr count: 40804
2021/12/14 01:20:51 Waiting for src bucket item count to get to: 50000 curr count: 40804
2021/12/14 01:21:01 Waiting for dst bucket item count to get to: 50000 curr count: 41088
2021/12/14 01:21:06 Waiting for dst bucket item count to get to: 50000 curr count: 41088
2021/12/14 01:21:11 Waiting for dst bucket item count to get to: 50000 curr count: 49304
2021/12/14 01:21:16 Waiting for dst bucket item count to get to: 50000 curr count: 49304
2021/12/14 01:21:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:21:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:21:16 Starting bucket ops to source bucket
2021/12/14 01:22:27 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:22:32 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:22:37 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:22:42 Waiting for dst bucket item count to get to: 0 curr count: 47387
2021/12/14 01:22:47 Waiting for dst bucket item count to get to: 0 curr count: 47387
2021/12/14 01:22:52 Waiting for dst bucket item count to get to: 0 curr count: 35914
2021/12/14 01:22:57 Waiting for dst bucket item count to get to: 0 curr count: 35914
2021/12/14 01:23:02 Waiting for dst bucket item count to get to: 0 curr count: 10715
2021/12/14 01:23:07 Waiting for dst bucket item count to get to: 0 curr count: 10715
2021/12/14 01:23:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:23:07 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:23:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:23:12 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 01:23:17 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 01:23:22 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 01:23:27 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/14 01:23:27 App: TestKVRebOutWithTimers got un-deployed
2021/12/14 01:23:27 No eventing-con process running
2021/12/14 01:23:27 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/14 01:23:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:23:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:23:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:23:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:23:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:23:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (270.01s)
=== RUN   TestKVSwapRebWithTimers
2021/12/14 01:23:51 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:23:51 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 01:23:51 No eventing-con process running
2021/12/14 01:23:51 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/14 01:23:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:23:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:23:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:23:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:24:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:24:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:24:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:24:09 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:24:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/14 01:24:15 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 01:24:20 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 01:24:25 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 01:24:30 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/14 01:24:31 App: TestKVSwapRebWithTimers got deployed
2021/12/14 01:24:31 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:24:31 Starting bucket ops to source bucket
2021/12/14 01:24:31 Removing node(s): [] from the cluster
2021/12/14 01:24:36 Rebalance progress: 12.5
2021/12/14 01:24:41 Rebalance progress: 12.5
2021/12/14 01:24:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:24:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 01:24:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:24:51 Rebalance progress: 12.5
2021/12/14 01:24:56 Rebalance progress: 12.5
2021/12/14 01:25:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:25:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:25:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:25:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 01:25:06 Rebalance progress: 12.5
2021/12/14 01:25:11 Rebalance progress: 12.5
2021/12/14 01:25:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:25:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:25:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:25:16 Waiting for src bucket item count to get to: 50000 curr count: 43141
2021/12/14 01:25:26 Waiting for dst bucket item count to get to: 50000 curr count: 40985
2021/12/14 01:25:31 Waiting for dst bucket item count to get to: 50000 curr count: 40985
2021/12/14 01:25:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:25:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:25:31 Starting bucket ops to source bucket
2021/12/14 01:26:42 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:26:47 Waiting for dst bucket item count to get to: 0 curr count: 48308
2021/12/14 01:26:52 Waiting for dst bucket item count to get to: 0 curr count: 48308
2021/12/14 01:26:57 Waiting for dst bucket item count to get to: 0 curr count: 37991
2021/12/14 01:27:02 Waiting for dst bucket item count to get to: 0 curr count: 37991
2021/12/14 01:27:07 Waiting for dst bucket item count to get to: 0 curr count: 37991
2021/12/14 01:27:12 Waiting for dst bucket item count to get to: 0 curr count: 17777
2021/12/14 01:27:17 Waiting for dst bucket item count to get to: 0 curr count: 17777
2021/12/14 01:27:22 Waiting for dst bucket item count to get to: 0 curr count: 4305
2021/12/14 01:27:27 Waiting for dst bucket item count to get to: 0 curr count: 4305
2021/12/14 01:27:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:27:27 Post DELETE ops. Expected item count: 0 got 0
2021/12/14 01:27:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:27:32 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:37 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:42 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:53 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:58 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/14 01:27:58 App: TestKVSwapRebWithTimers got un-deployed
2021/12/14 01:27:58 No eventing-con process running
2021/12/14 01:27:58 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/14 01:27:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:28:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:28:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:28:06 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:28:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:28:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (270.45s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/14 01:28:21 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:28:21 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 01:28:21 No eventing-con process running
2021/12/14 01:28:21 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/14 01:28:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:28:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:28:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:28:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:28:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:28:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:28:40 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:28:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 01:28:46 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 01:28:51 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 01:28:56 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 01:29:01 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/14 01:29:01 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/14 01:29:01 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:29:01 Starting bucket ops to source bucket
2021/12/14 01:29:02 Removing node(s): [] from the cluster
2021/12/14 01:29:07 Rebalance progress: 12.5
2021/12/14 01:29:12 Rebalance progress: 12.5
2021/12/14 01:29:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:29:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:29:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:29:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:29:23 Rebalance progress: 12.5
2021/12/14 01:29:28 Rebalance progress: 12.5
2021/12/14 01:29:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:29:33 Waiting for src bucket item count to get to: 50000 curr count: 36134
2021/12/14 01:29:38 Waiting for src bucket item count to get to: 50000 curr count: 36134
2021/12/14 01:29:43 Waiting for src bucket item count to get to: 50000 curr count: 45075
2021/12/14 01:29:48 Waiting for src bucket item count to get to: 50000 curr count: 45075
2021/12/14 01:29:58 Waiting for dst bucket item count to get to: 50000 curr count: 29113
2021/12/14 01:30:03 Waiting for dst bucket item count to get to: 50000 curr count: 29113
2021/12/14 01:30:08 Waiting for dst bucket item count to get to: 50000 curr count: 42302
2021/12/14 01:30:13 Waiting for dst bucket item count to get to: 50000 curr count: 42302
2021/12/14 01:30:18 Waiting for dst bucket item count to get to: 50000 curr count: 49387
2021/12/14 01:30:23 Waiting for dst bucket item count to get to: 50000 curr count: 49387
2021/12/14 01:30:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:30:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:30:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:30:28 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:33 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:38 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:43 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:48 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:53 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/14 01:30:53 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/14 01:30:53 No eventing-con process running
2021/12/14 01:30:53 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/14 01:30:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:31:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:31:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:31:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:31:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:31:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:31:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:31:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (185.53s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/14 01:31:22 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:31:22 Removing node(s): [] from the cluster
2021/12/14 01:31:27 Rebalance progress: 100
2021/12/14 01:31:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:31:32 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 01:31:32 No eventing-con process running
2021/12/14 01:31:32 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/14 01:31:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:31:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:31:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:31:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:31:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:31:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:31:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:31:50 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:31:52 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 01:31:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 01:32:02 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 01:32:07 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 01:32:12 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/14 01:32:12 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/14 01:32:12 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:32:12 Starting bucket ops to source bucket
2021/12/14 01:32:17 Rebalance progress: 12.5
2021/12/14 01:32:22 Rebalance progress: 12.5
2021/12/14 01:32:27 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:32:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:32:27 Waiting for src bucket item count to get to: 50000 curr count: 9756
2021/12/14 01:32:32 Waiting for src bucket item count to get to: 50000 curr count: 23528
2021/12/14 01:32:37 Waiting for src bucket item count to get to: 50000 curr count: 23528
2021/12/14 01:32:42 Waiting for src bucket item count to get to: 50000 curr count: 37197
2021/12/14 01:32:47 Waiting for src bucket item count to get to: 50000 curr count: 37197
2021/12/14 01:32:52 Waiting for src bucket item count to get to: 50000 curr count: 46828
2021/12/14 01:32:57 Waiting for src bucket item count to get to: 50000 curr count: 46828
2021/12/14 01:33:02 Waiting for src bucket item count to get to: 50000 curr count: 46828
2021/12/14 01:33:12 Waiting for dst bucket item count to get to: 50000 curr count: 28007
2021/12/14 01:33:17 Waiting for dst bucket item count to get to: 50000 curr count: 28007
2021/12/14 01:33:22 Waiting for dst bucket item count to get to: 50000 curr count: 44343
2021/12/14 01:33:27 Waiting for dst bucket item count to get to: 50000 curr count: 44343
2021/12/14 01:33:27 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:33:27 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:33:27 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:33:32 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:37 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:42 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:47 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:52 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:57 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/14 01:33:57 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/14 01:33:57 No eventing-con process running
2021/12/14 01:33:57 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/14 01:33:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:34:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:34:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:34:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:34:11 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:34:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:34:21 Waiting for dst bucket item count to get to: 0 curr count: 41384
2021/12/14 01:34:26 Waiting for dst bucket item count to get to: 0 curr count: 41384
2021/12/14 01:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (189.12s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/14 01:34:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:34:36 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 01:34:36 No eventing-con process running
2021/12/14 01:34:36 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/14 01:34:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:34:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:34:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:34:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:34:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:34:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:34:54 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:34:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 01:35:01 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 01:35:06 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 01:35:11 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 01:35:16 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/14 01:35:16 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/14 01:35:16 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:35:16 Starting bucket ops to source bucket
2021/12/14 01:35:16 Removing node(s): [] from the cluster
2021/12/14 01:35:21 Rebalance progress: 12.5
2021/12/14 01:35:26 Rebalance progress: 12.5
2021/12/14 01:35:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:35:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:35:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 01:35:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:35:36 Rebalance progress: 12.5
2021/12/14 01:35:41 Rebalance progress: 12.5
2021/12/14 01:35:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:35:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:35:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 01:35:51 Rebalance progress: 12.5
2021/12/14 01:35:56 Rebalance progress: 12.5
2021/12/14 01:36:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:36:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:36:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:36:02 Waiting for src bucket item count to get to: 50000 curr count: 48646
2021/12/14 01:36:07 Waiting for src bucket item count to get to: 50000 curr count: 48646
2021/12/14 01:36:17 Waiting for dst bucket item count to get to: 50000 curr count: 37677
2021/12/14 01:36:22 Waiting for dst bucket item count to get to: 50000 curr count: 37677
2021/12/14 01:36:22 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:36:22 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:36:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:36:27 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:32 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:37 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:42 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:47 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:52 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/14 01:36:52 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/14 01:36:52 No eventing-con process running
2021/12/14 01:36:52 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/14 01:36:54 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:36:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:37:04 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:37:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:37:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:37:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:37:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:37:20 Waiting for dst bucket item count to get to: 0 curr count: 27267
2021/12/14 01:37:25 Waiting for dst bucket item count to get to: 0 curr count: 27267
2021/12/14 01:37:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (179.67s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/14 01:37:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:37:36 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 01:37:36 No eventing-con process running
2021/12/14 01:37:36 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/14 01:37:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:37:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:37:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:37:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:37:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:37:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:37:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:37:54 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:37:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 01:38:00 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 01:38:05 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 01:38:10 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 01:38:15 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/14 01:38:15 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/14 01:38:15 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:38:15 Starting bucket ops to source bucket
2021/12/14 01:38:16 Removing node(s): [] from the cluster
2021/12/14 01:38:21 Rebalance progress: 12.5
2021/12/14 01:38:26 Rebalance progress: 12.5
2021/12/14 01:38:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:38:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:38:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:38:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:38:36 Rebalance progress: 12.5
2021/12/14 01:38:41 Rebalance progress: 12.5
2021/12/14 01:38:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:38:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:38:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:38:46 Waiting for src bucket item count to get to: 50000 curr count: 31478
2021/12/14 01:38:51 Waiting for src bucket item count to get to: 50000 curr count: 43003
2021/12/14 01:38:56 Waiting for src bucket item count to get to: 50000 curr count: 43003
2021/12/14 01:39:06 Waiting for dst bucket item count to get to: 50000 curr count: 27414
2021/12/14 01:39:11 Waiting for dst bucket item count to get to: 50000 curr count: 27414
2021/12/14 01:39:16 Waiting for dst bucket item count to get to: 50000 curr count: 36135
2021/12/14 01:39:21 Waiting for dst bucket item count to get to: 50000 curr count: 36135
2021/12/14 01:39:21 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:39:21 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:39:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:39:26 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:31 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:36 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:41 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:46 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:51 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/14 01:39:51 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/14 01:39:51 No eventing-con process running
2021/12/14 01:39:51 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/14 01:39:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:39:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:40:03 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:40:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:40:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:40:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:40:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:40:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (169.20s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/14 01:40:20 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:40:20 Removing node(s): [] from the cluster
2021/12/14 01:40:25 Rebalance progress: 100
2021/12/14 01:40:30 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:40:30 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 01:40:30 No eventing-con process running
2021/12/14 01:40:30 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/14 01:40:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:40:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:40:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:40:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:40:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:40:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:40:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:40:49 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:40:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 01:40:55 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 01:41:00 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 01:41:05 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 01:41:10 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/14 01:41:10 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/14 01:41:10 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:41:10 Starting bucket ops to source bucket
2021/12/14 01:41:15 Rebalance progress: 12.5
2021/12/14 01:41:20 Rebalance progress: 12.5
2021/12/14 01:41:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:41:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:41:25 Waiting for src bucket item count to get to: 50000 curr count: 11695
2021/12/14 01:41:30 Waiting for src bucket item count to get to: 50000 curr count: 25758
2021/12/14 01:41:35 Waiting for src bucket item count to get to: 50000 curr count: 25758
2021/12/14 01:41:40 Waiting for src bucket item count to get to: 50000 curr count: 38130
2021/12/14 01:41:45 Waiting for src bucket item count to get to: 50000 curr count: 38130
2021/12/14 01:41:50 Waiting for src bucket item count to get to: 50000 curr count: 47791
2021/12/14 01:41:55 Waiting for src bucket item count to get to: 50000 curr count: 47791
2021/12/14 01:42:00 Waiting for src bucket item count to get to: 50000 curr count: 47791
2021/12/14 01:42:10 Waiting for dst bucket item count to get to: 50000 curr count: 31935
2021/12/14 01:42:15 Waiting for dst bucket item count to get to: 50000 curr count: 31935
2021/12/14 01:42:20 Waiting for dst bucket item count to get to: 50000 curr count: 43997
2021/12/14 01:42:25 Waiting for dst bucket item count to get to: 50000 curr count: 43997
2021/12/14 01:42:25 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:42:25 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:42:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:42:30 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:35 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:41 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:46 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:51 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:56 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/14 01:42:56 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/14 01:42:56 No eventing-con process running
2021/12/14 01:42:56 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/14 01:42:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:43:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:43:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:43:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:43:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:43:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:43:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (184.63s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/14 01:43:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:43:29 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 01:43:29 No eventing-con process running
2021/12/14 01:43:29 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/14 01:43:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:43:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:43:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:43:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:43:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:43:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:43:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:43:48 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:43:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 01:43:54 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 01:43:59 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 01:44:04 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 01:44:09 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/14 01:44:09 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/14 01:44:09 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:44:09 Starting bucket ops to source bucket
2021/12/14 01:44:09 Removing node(s): [] from the cluster
2021/12/14 01:44:14 Rebalance progress: 12.5
2021/12/14 01:44:19 Rebalance progress: 12.5
2021/12/14 01:44:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:44:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 01:44:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:44:30 Rebalance progress: 12.5
2021/12/14 01:44:35 Rebalance progress: 12.5
2021/12/14 01:44:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:44:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 01:44:45 Rebalance progress: 12.5
2021/12/14 01:44:50 Rebalance progress: 12.5
2021/12/14 01:44:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:44:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:44:55 Waiting for src bucket item count to get to: 50000 curr count: 46083
2021/12/14 01:45:00 Waiting for src bucket item count to get to: 50000 curr count: 46083
2021/12/14 01:45:10 Waiting for dst bucket item count to get to: 50000 curr count: 36411
2021/12/14 01:45:15 Waiting for dst bucket item count to get to: 50000 curr count: 36411
2021/12/14 01:45:20 Waiting for dst bucket item count to get to: 50000 curr count: 43593
2021/12/14 01:45:20 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:45:20 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:45:20 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:45:25 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:30 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:35 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:40 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/14 01:45:50 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/14 01:45:50 No eventing-con process running
2021/12/14 01:45:50 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/14 01:45:52 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:45:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:46:02 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:46:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:46:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:46:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:46:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:46:19 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:46:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (179.71s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/14 01:46:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:46:29 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 01:46:29 No eventing-con process running
2021/12/14 01:46:29 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/14 01:46:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:46:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:46:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:46:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:46:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:46:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:46:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:46:48 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:46:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 01:46:54 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 01:46:59 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 01:47:04 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 01:47:09 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/14 01:47:09 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/14 01:47:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:47:09 Starting bucket ops to source bucket
2021/12/14 01:47:09 Removing node(s): [] from the cluster
2021/12/14 01:47:14 Rebalance progress: 12.5
2021/12/14 01:47:19 Rebalance progress: 12.5
2021/12/14 01:47:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:47:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:47:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:47:29 Rebalance progress: 12.5
2021/12/14 01:47:34 Rebalance progress: 12.5
2021/12/14 01:47:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:47:39 Waiting for src bucket item count to get to: 50000 curr count: 24428
2021/12/14 01:47:44 Waiting for src bucket item count to get to: 50000 curr count: 34380
2021/12/14 01:47:49 Waiting for src bucket item count to get to: 50000 curr count: 34380
2021/12/14 01:47:54 Waiting for src bucket item count to get to: 50000 curr count: 47073
2021/12/14 01:47:59 Waiting for src bucket item count to get to: 50000 curr count: 47073
2021/12/14 01:48:10 Waiting for dst bucket item count to get to: 50000 curr count: 48270
2021/12/14 01:48:15 Waiting for dst bucket item count to get to: 50000 curr count: 48270
2021/12/14 01:48:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:48:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:48:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:48:20 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:25 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:30 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:35 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:40 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:45 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/14 01:48:45 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/14 01:48:45 No eventing-con process running
2021/12/14 01:48:45 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/14 01:48:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:48:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:48:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:48:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:48:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:49:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:49:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:49:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (169.35s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/14 01:49:13 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:49:14 Removing node(s): [] from the cluster
2021/12/14 01:49:19 Rebalance progress: 100
2021/12/14 01:49:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:49:24 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 01:49:24 No eventing-con process running
2021/12/14 01:49:24 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/14 01:49:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:49:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:49:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:49:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:49:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:49:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:49:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:49:42 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:49:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 01:49:48 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 01:49:53 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 01:49:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 01:50:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/14 01:50:03 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/14 01:50:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:50:03 Starting bucket ops to source bucket
2021/12/14 01:50:09 Rebalance progress: 12.5
2021/12/14 01:50:14 Rebalance progress: 12.5
2021/12/14 01:50:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:50:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:50:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:50:19 Waiting for src bucket item count to get to: 50000 curr count: 7491
2021/12/14 01:50:24 Waiting for src bucket item count to get to: 50000 curr count: 19661
2021/12/14 01:50:29 Waiting for src bucket item count to get to: 50000 curr count: 19661
2021/12/14 01:50:34 Waiting for src bucket item count to get to: 50000 curr count: 30455
2021/12/14 01:50:39 Waiting for src bucket item count to get to: 50000 curr count: 30455
2021/12/14 01:50:44 Waiting for src bucket item count to get to: 50000 curr count: 30455
2021/12/14 01:50:49 Waiting for src bucket item count to get to: 50000 curr count: 41688
2021/12/14 01:50:54 Waiting for src bucket item count to get to: 50000 curr count: 41688
2021/12/14 01:51:04 Waiting for dst bucket item count to get to: 50000 curr count: 40674
2021/12/14 01:51:09 Waiting for dst bucket item count to get to: 50000 curr count: 40674
2021/12/14 01:51:09 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:51:09 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:51:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:51:14 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:19 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:24 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:29 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:34 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/14 01:51:39 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/14 01:51:39 No eventing-con process running
2021/12/14 01:51:39 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/14 01:51:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:51:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:51:51 Waiting for dst bucket item count to get to: 0 curr count: 38342
2021/12/14 01:51:56 Waiting for dst bucket item count to get to: 0 curr count: 38342
2021/12/14 01:51:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:51:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:52:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:52:08 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:52:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.17s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/14 01:52:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:52:18 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 01:52:18 No eventing-con process running
2021/12/14 01:52:18 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/14 01:52:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:52:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:52:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:52:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:52:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:52:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:52:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:52:36 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:52:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 01:52:42 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 01:52:47 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 01:52:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 01:52:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/14 01:52:57 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/14 01:52:57 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/14 01:52:57 Starting bucket ops to source bucket
2021/12/14 01:52:57 Removing node(s): [] from the cluster
2021/12/14 01:53:03 Rebalance progress: 12.5
2021/12/14 01:53:08 Rebalance progress: 12.5
2021/12/14 01:53:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:53:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 01:53:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:53:18 Rebalance progress: 12.5
2021/12/14 01:53:23 Rebalance progress: 12.5
2021/12/14 01:53:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:53:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 01:53:33 Rebalance progress: 12.5
2021/12/14 01:53:38 Rebalance progress: 12.5
2021/12/14 01:53:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:53:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:53:43 Waiting for src bucket item count to get to: 50000 curr count: 47045
2021/12/14 01:53:48 Waiting for src bucket item count to get to: 50000 curr count: 47045
2021/12/14 01:53:58 Waiting for dst bucket item count to get to: 50000 curr count: 38447
2021/12/14 01:54:03 Waiting for dst bucket item count to get to: 50000 curr count: 38447
2021/12/14 01:54:08 Waiting for dst bucket item count to get to: 50000 curr count: 46573
2021/12/14 01:54:13 Waiting for dst bucket item count to get to: 50000 curr count: 46573
2021/12/14 01:54:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:54:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:54:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:54:18 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:23 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/14 01:54:44 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/14 01:54:44 No eventing-con process running
2021/12/14 01:54:44 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/14 01:54:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:54:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:54:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:54:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:55:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:55:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:55:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (179.72s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/14 01:55:17 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:55:17 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 01:55:17 No eventing-con process running
2021/12/14 01:55:17 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/14 01:55:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:55:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:55:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:55:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:55:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:55:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:55:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:55:36 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:55:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/14 01:55:42 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 01:55:47 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 01:55:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 01:55:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/14 01:55:57 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/14 01:55:57 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:55:57 Starting bucket ops to source bucket
2021/12/14 01:55:57 Removing node(s): [] from the cluster
2021/12/14 01:56:03 Rebalance progress: 12.5
2021/12/14 01:56:08 Rebalance progress: 12.5
2021/12/14 01:56:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:56:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:56:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:56:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:56:18 Rebalance progress: 12.5
2021/12/14 01:56:23 Rebalance progress: 12.5
2021/12/14 01:56:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:56:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:56:28 Waiting for src bucket item count to get to: 50000 curr count: 27111
2021/12/14 01:56:33 Waiting for src bucket item count to get to: 50000 curr count: 37719
2021/12/14 01:56:38 Waiting for src bucket item count to get to: 50000 curr count: 37719
2021/12/14 01:56:43 Waiting for src bucket item count to get to: 50000 curr count: 49366
2021/12/14 01:56:48 Waiting for src bucket item count to get to: 50000 curr count: 49366
2021/12/14 01:56:58 Waiting for dst bucket item count to get to: 50000 curr count: 44417
2021/12/14 01:57:03 Waiting for dst bucket item count to get to: 50000 curr count: 44417
2021/12/14 01:57:03 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:57:03 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:57:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 01:57:08 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:13 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:33 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/14 01:57:33 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/14 01:57:33 No eventing-con process running
2021/12/14 01:57:33 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/14 01:57:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:57:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:57:45 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:57:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:57:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:57:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:57:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 01:57:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.40s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/14 01:58:02 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 01:58:02 Removing node(s): [] from the cluster
2021/12/14 01:58:07 Rebalance progress: 100
2021/12/14 01:58:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 01:58:12 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 01:58:12 No eventing-con process running
2021/12/14 01:58:12 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/14 01:58:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:58:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 01:58:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:58:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:58:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:58:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 01:58:31 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/14 01:58:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/14 01:58:37 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 01:58:42 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 01:58:47 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 01:58:52 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/14 01:58:52 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/14 01:58:52 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 01:58:52 Starting bucket ops to source bucket
2021/12/14 01:58:57 Rebalance progress: 12.5
2021/12/14 01:59:02 Rebalance progress: 12.5
2021/12/14 01:59:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 01:59:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 01:59:07 Waiting for src bucket item count to get to: 50000 curr count: 9744
2021/12/14 01:59:12 Waiting for src bucket item count to get to: 50000 curr count: 22312
2021/12/14 01:59:17 Waiting for src bucket item count to get to: 50000 curr count: 22312
2021/12/14 01:59:22 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/14 01:59:27 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/14 01:59:32 Waiting for src bucket item count to get to: 50000 curr count: 33653
2021/12/14 01:59:37 Waiting for src bucket item count to get to: 50000 curr count: 44106
2021/12/14 01:59:42 Waiting for src bucket item count to get to: 50000 curr count: 44106
2021/12/14 01:59:52 Waiting for dst bucket item count to get to: 50000 curr count: 37584
2021/12/14 01:59:57 Waiting for dst bucket item count to get to: 50000 curr count: 37584
2021/12/14 01:59:57 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 01:59:57 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 01:59:57 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:00:02 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:07 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:12 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:17 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:22 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:27 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/14 02:00:27 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/14 02:00:27 No eventing-con process running
2021/12/14 02:00:27 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/14 02:00:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:00:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:00:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 02:00:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:00:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:00:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:00:51 Waiting for dst bucket item count to get to: 0 curr count: 48643
2021/12/14 02:00:56 Waiting for dst bucket item count to get to: 0 curr count: 48643
2021/12/14 02:00:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (179.33s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/14 02:01:06 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:01:06 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 02:01:06 No eventing-con process running
2021/12/14 02:01:06 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/14 02:01:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:01:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:01:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:01:15 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:01:20 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:01:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:01:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:01:25 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:01:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/14 02:01:31 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 02:01:36 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 02:01:41 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 02:01:46 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/14 02:01:46 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/14 02:01:46 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:01:46 Starting bucket ops to source bucket
2021/12/14 02:01:46 Removing node(s): [] from the cluster
2021/12/14 02:01:51 Rebalance progress: 12.5
2021/12/14 02:01:56 Rebalance progress: 12.5
2021/12/14 02:02:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:02:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:02:01 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:02:07 Rebalance progress: 12.5
2021/12/14 02:02:12 Rebalance progress: 12.5
2021/12/14 02:02:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:02:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:02:22 Rebalance progress: 12.5
2021/12/14 02:02:27 Rebalance progress: 12.5
2021/12/14 02:02:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:02:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:02:32 Waiting for src bucket item count to get to: 50000 curr count: 48278
2021/12/14 02:02:37 Waiting for src bucket item count to get to: 50000 curr count: 48278
2021/12/14 02:02:47 Waiting for dst bucket item count to get to: 50000 curr count: 44390
2021/12/14 02:02:52 Waiting for dst bucket item count to get to: 50000 curr count: 44390
2021/12/14 02:02:52 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/14 02:02:52 Post SET ops. Expected item count: 50000 got 50000
2021/12/14 02:02:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:02:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:12 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:17 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:22 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/14 02:03:22 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/14 02:03:22 No eventing-con process running
2021/12/14 02:03:22 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/14 02:03:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:03:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:03:34 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 02:03:39 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/14 02:03:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:03:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:03:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:03:51 Waiting for dst bucket item count to get to: 0 curr count: 34197
2021/12/14 02:03:56 Waiting for dst bucket item count to get to: 0 curr count: 34197
2021/12/14 02:03:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (179.66s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/14 02:04:06 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:04:06 App: bucket_op_on_update got un-deployed
2021/12/14 02:04:06 No eventing-con process running
2021/12/14 02:04:06 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:04:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:04:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:04:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:04:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:04:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:04:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:04:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:04:24 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:04:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:04:30 Starting bucket ops to source bucket
2021/12/14 02:04:35 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:04:40 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:04:45 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:04:45 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:04:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:04:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:04:46 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:04:51 Rebalance progress: 12.5
2021/12/14 02:04:56 Rebalance progress: 12.5
2021/12/14 02:05:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:05:01 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:05:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:05:06 Rebalance progress: 12.5
2021/12/14 02:05:11 Rebalance progress: 12.5
2021/12/14 02:05:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:05:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 02:05:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:05:21 Rebalance progress: 12.5
2021/12/14 02:05:26 Rebalance progress: 12.5
2021/12/14 02:05:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:05:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:05:37 Rebalance progress: 12.5
2021/12/14 02:05:42 Rebalance progress: 12.5
2021/12/14 02:05:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:05:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:05:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:05:52 Rebalance progress: 12.5
2021/12/14 02:05:57 Rebalance progress: 12.5
2021/12/14 02:06:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:06:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:06:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:06:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 02:06:07 Rebalance progress: 12.5
2021/12/14 02:06:12 Rebalance progress: 12.5
2021/12/14 02:06:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:06:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:06:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:06:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:06:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:06:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:06:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:06:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:06:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:06:42 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:06:42 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:06:42 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:06:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:06:49 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:06:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10630
2021/12/14 02:06:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10630
2021/12/14 02:06:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:07:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:07:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:07:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8159
2021/12/14 02:07:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8159
2021/12/14 02:07:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.09s)
=== RUN   TestKVForceFailover
2021/12/14 02:07:31 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:07:31 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:07:31 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:07:31 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:07:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:07:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:07:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:07:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:07:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:07:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:07:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:07:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:07:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:07:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:07:50 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:07:51 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:07:51 Starting bucket ops to source bucket
2021/12/14 02:07:56 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:08:01 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:08:06 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:08:11 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:08:11 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:08:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:08:11 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 02:08:31 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 02:08:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/14 02:08:36 Rebalance progress: 12.5
2021/12/14 02:08:41 Rebalance progress: 12.5
2021/12/14 02:08:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:08:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:08:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:08:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/14 02:08:51 Rebalance progress: 12.5
2021/12/14 02:08:56 Rebalance progress: 12.5
2021/12/14 02:09:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:09:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:09:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:09:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:09:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:09:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:09:17 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:09:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:09:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:09:27 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:09:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:09:27 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:09:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:09:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:09:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7034
2021/12/14 02:09:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:09:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:09:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:09:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3868
2021/12/14 02:09:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3868
2021/12/14 02:09:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (159.70s)
=== RUN   TestKVHardFailoverRecovery
2021/12/14 02:10:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:10:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:10:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:10:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:10:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:10:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:10:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:10:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:10:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:10:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:10:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:10:24 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:10:25 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:10:30 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:10:35 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:10:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:10:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:10:45 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:10:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:10:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:10:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:10:45 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:10:51 Rebalance progress: 12.5
2021/12/14 02:10:56 Rebalance progress: 12.5
2021/12/14 02:11:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:11:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:11:01 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/14 02:12:01 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/14 02:12:06 Rebalance progress: 12.5
2021/12/14 02:12:11 Rebalance progress: 12.5
2021/12/14 02:12:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:12:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:12:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:12:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:12:21 Rebalance progress: 12.5
2021/12/14 02:12:26 Rebalance progress: 12.5
2021/12/14 02:12:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:12:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:12:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:12:31 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:12:36 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:12:41 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:12:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:12:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:12:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:13:01 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:13:01 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:13:01 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:13:01 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:13:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:13:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:13:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12038
2021/12/14 02:13:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12038
2021/12/14 02:13:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:13:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:13:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:13:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.28s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/14 02:13:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:13:35 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:13:35 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:13:35 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:13:37 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:13:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:13:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:13:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:13:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:13:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:13:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:13:53 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:13:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:13:59 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:14:04 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:14:09 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:14:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:14:14 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:14:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:14:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:14:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 02:14:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:14:20 Rebalance progress: 12.5
2021/12/14 02:14:25 Rebalance progress: 12.5
2021/12/14 02:14:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:14:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/14 02:14:35 Rebalance progress: 12.5
2021/12/14 02:14:40 Rebalance progress: 12.5
2021/12/14 02:14:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:14:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:14:45 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:14:50 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:14:55 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:00 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:16 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:15:16 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:15:16 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:15:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:15:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:15:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4564
2021/12/14 02:15:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:15:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:15:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:15:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4564
2021/12/14 02:15:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (134.47s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/14 02:15:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:15:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:15:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:15:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:15:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:15:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:15:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:15:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:16:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:16:03 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:16:04 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:16:09 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:16:14 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:16:19 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:16:24 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:16:24 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:16:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:16:24 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:16:29 Rebalance progress: 12.5
2021/12/14 02:16:34 Rebalance progress: 12.5
2021/12/14 02:16:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:16:39 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:16:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:16:45 Rebalance progress: 12.5
2021/12/14 02:16:50 Rebalance progress: 12.5
2021/12/14 02:16:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:16:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:16:55 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:17:00 Rebalance progress: 12.5
2021/12/14 02:17:05 Rebalance progress: 12.5
2021/12/14 02:17:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:17:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:17:10 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:17:15 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:20 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:25 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:30 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:35 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:40 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:17:40 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:17:40 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:17:40 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:17:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:17:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:17:52 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6093
2021/12/14 02:17:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6093
2021/12/14 02:17:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:17:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:18:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:18:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6093
2021/12/14 02:18:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:18:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:18:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:18:14 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:18:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:18:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:18:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:18:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:18:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:18:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:18:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.43s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/14 02:18:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:18:38 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:18:38 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:18:38 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:18:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:18:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:18:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:18:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:18:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:18:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:18:56 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:18:56 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 02:18:56 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:18:56 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 02:18:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:19:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:19:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:19:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:19:10 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:19:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:19:11 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:19:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 02:19:17 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:19:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:19:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:19:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 02:19:32 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:19:37 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:19:42 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:19:47 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:19:47 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:19:47 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:19:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:19:47 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:19:52 Rebalance progress: 12.5
2021/12/14 02:19:57 Rebalance progress: 12.5
2021/12/14 02:20:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:20:03 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:20:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:20:08 Rebalance progress: 12.5
2021/12/14 02:20:13 Rebalance progress: 12.5
2021/12/14 02:20:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:20:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/14 02:20:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:20:23 Rebalance progress: 12.5
2021/12/14 02:20:28 Rebalance progress: 12.5
2021/12/14 02:20:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:20:33 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:20:38 Rebalance progress: 12.5
2021/12/14 02:20:43 Rebalance progress: 12.5
2021/12/14 02:20:48 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:20:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:20:48 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:20:54 Rebalance progress: 12.5
2021/12/14 02:20:59 Rebalance progress: 12.5
2021/12/14 02:21:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:21:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:21:04 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/14 02:21:09 Rebalance progress: 12.5
2021/12/14 02:21:14 Rebalance progress: 12.5
2021/12/14 02:21:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:21:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:21:19 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:21:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:21:49 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:21:49 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:21:49 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:21:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:21:59 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:22:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11697
2021/12/14 02:22:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:22:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:22:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:22:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12152
2021/12/14 02:22:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:22:16 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:22:21 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:22:26 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:22:31 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:22:36 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:22:41 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:22:41 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 02:22:41 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:22:42 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 02:22:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:22:48 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:22:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:22:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:22:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:22:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (267.42s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/14 02:23:05 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:23:05 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:23:05 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:23:05 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:23:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:23:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:23:24 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:23:24 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 02:23:24 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:23:24 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/14 02:23:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:23:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:37 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:23:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:23:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:23:38 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:23:39 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/14 02:23:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:23:49 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:23:54 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:23:59 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:23:59 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/14 02:24:04 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:24:09 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:24:14 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/14 02:24:14 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:24:15 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:24:15 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:24:20 Rebalance progress: 12.5
2021/12/14 02:24:25 Rebalance progress: 12.5
2021/12/14 02:24:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:24:30 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:24:31 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:24:36 Rebalance progress: 12.5
2021/12/14 02:24:41 Rebalance progress: 12.5
2021/12/14 02:24:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:24:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:24:46 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:24:51 Rebalance progress: 12.5
2021/12/14 02:24:56 Rebalance progress: 12.5
2021/12/14 02:25:01 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:25:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:25:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:25:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:25:06 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:25:11 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:25:16 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:25:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:25:26 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:25:26 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:25:26 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:25:26 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:25:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:25:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:25:40 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 201
2021/12/14 02:25:45 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 201
2021/12/14 02:25:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:25:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:25:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:25:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7631
2021/12/14 02:25:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:25:57 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:26:02 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:07 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:12 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:17 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:22 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:27 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/14 02:26:27 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/14 02:26:27 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:26:27 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/14 02:26:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:26:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:26:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:26:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:26:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:26:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (225.87s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/14 02:26:51 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:26:51 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:26:51 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:26:51 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:26:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:26:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:26:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:27:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:27:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:27:10 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:27:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:27:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:27:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:27:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:27:31 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:27:31 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:27:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:27:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:27:51 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:27:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:28:01 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:28:06 Rebalance progress: 12.5
2021/12/14 02:28:11 Rebalance progress: 12.5
2021/12/14 02:28:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:28:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:28:17 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:28:22 Rebalance progress: 12.5
2021/12/14 02:28:27 Rebalance progress: 12.5
2021/12/14 02:28:32 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:28:32 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:28:37 Rebalance progress: 12.5
2021/12/14 02:28:42 Rebalance progress: 12.5
2021/12/14 02:28:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:28:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:28:47 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:28:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:28:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:29:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:29:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:29:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:29:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:29:12 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:29:13 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:29:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:29:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:29:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9124
2021/12/14 02:29:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:29:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:29:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:29:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9124
2021/12/14 02:29:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (175.16s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/14 02:29:46 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:29:46 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:29:46 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:29:46 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/14 02:29:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:29:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:29:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:29:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:00 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:30:00 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:30:05 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:30:06 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:30:11 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/14 02:30:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:30:21 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:30:26 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:30:26 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:30:26 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:30:26 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:30:31 Rebalance progress: 12.5
2021/12/14 02:30:36 Rebalance progress: 12.5
2021/12/14 02:30:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:30:41 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:30:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:30:51 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:30:56 Rebalance progress: 12.5
2021/12/14 02:31:01 Rebalance progress: 12.5
2021/12/14 02:31:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:31:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/14 02:31:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/14 02:31:12 Rebalance progress: 12.5
2021/12/14 02:31:17 Rebalance progress: 12.5
2021/12/14 02:31:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:31:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/14 02:31:27 Rebalance progress: 12.5
2021/12/14 02:31:32 Rebalance progress: 12.5
2021/12/14 02:31:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:31:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:31:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:31:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:31:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:31:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:31:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:32:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:32:08 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:32:08 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:32:08 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:32:08 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:32:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:32:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:32:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:32:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:32:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:32:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8633
2021/12/14 02:32:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (170.07s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/14 02:32:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/14 02:32:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/14 02:32:37 Rebalance progress: 100
2021/12/14 02:32:37 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:33:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:33:07 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/14 02:33:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/14 02:33:13 Removing node(s): [] from the cluster
2021/12/14 02:33:13 Starting bucket ops to source bucket
2021/12/14 02:33:18 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/14 02:33:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:33:28 Waiting for app: bucket_op_on_update to get deployed
2021/12/14 02:33:28 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:33:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/14 02:33:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/14 02:33:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/14 02:33:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/14 02:33:58 eventingSetup.go:46: No eventing-con process running
2021/12/14 02:33:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/14 02:34:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:34:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:34:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 199
2021/12/14 02:34:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 199
2021/12/14 02:34:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/14 02:34:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/14 02:34:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/14 02:34:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1505
2021/12/14 02:34:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (120.73s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12112.156s

Rebalance tests took 3 hours, 21 minutes and 54 seconds.

Testing Succeeded


Version: versions-13.12.2021-18.53.cfg
Build Log: make-13.12.2021-18.53.log
Server Log: logs-13.12.2021-18.53.tar.gz

Finished