Building

Started building at 2021/12/04 11:15:40
Calculating base
Updating mirror
Basing run on 7.1.0-1840 c619f8890
Updating tree for run 04.12.2021-11.15
gocb is at 635fa6c, changes since last good build: none
libcouchbase is at f6822fc, changes since last good build: none
indexing is at a540074, changes since last good build: none
kv_engine is at 7e709d9, changes since last good build: none
ns_server is at bea2029, changes since last good build: 
 bea202977 MB-49887 Return the most recent event logs when no query params ...
couchstore is at 5aa7c4c, changes since last good build: none
couchdb is at fca2e6b, changes since last good build: none
testrunner is at cc2f3d8, changes since last good build: 
 cc2f3d87c Add TC for n1ql js udf
Switching eventing to unstable
eventing is at 9fce322, changes since last good build: none
Switching eventing-ee to unstable
eventing-ee is at 7060ef8, changes since last good build: none
Pulling in uncommitted change 160397 at refs/changes/97/160397/3
Total 23 (delta 15), reused 17 (delta 15)
[unstable 3af89d69] MB-100 : Do not fail n1ql timeout test if we are catching timeout exceptions successfully
 Author: abhijpes 
 Date: Mon Aug 30 08:24:38 2021 +0530
 1 file changed, 2 insertions(+), 2 deletions(-)
Pulling in uncommitted change 163449 at refs/changes/49/163449/1
Total 48 (delta 27), reused 39 (delta 27)
[unstable 57e60c91] MB-100: collect data when test fails
 Author: AnkitPrabhu 
 Date: Tue Oct 12 14:02:31 2021 +0530
 1 file changed, 2 insertions(+)
Pulling in uncommitted change 165976 at refs/changes/76/165976/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 4b5e31dc] MB-48538: Add new audit ids for different events
 Author: AnkitPrabhu 
 Date: Tue Nov 23 14:29:10 2021 +0530
 1 file changed, 337 insertions(+), 61 deletions(-)
Pulling in uncommitted change 166697 at refs/changes/97/166697/4
Total 10 (delta 9), reused 10 (delta 9)
[unstable 703a428e] MB-49547: Check bucket instance to get kv node for dcp request
 Author: AnkitPrabhu 
 Date: Thu Dec 2 12:22:17 2021 +0530
 6 files changed, 38 insertions(+), 109 deletions(-)
Pulling in uncommitted change 166698 at refs/changes/98/166698/3
Total 4 (delta 3), reused 4 (delta 3)
[unstable 4a71312f] MB-48334 : Fail rebalance unless all functions deployed on current encryption level
 Author: abhijpes 
 Date: Wed Dec 1 14:51:29 2021 +0530
 1 file changed, 9 insertions(+)
Pulling in uncommitted change 166812 at refs/changes/12/166812/5
Total 24 (delta 20), reused 24 (delta 20)
[unstable 69eb52bb] MB-49547: Refresh bucket info from streaming endpoint instance
 Author: AnkitPrabhu 
 Date: Thu Dec 2 13:55:55 2021 +0530
 7 files changed, 147 insertions(+), 235 deletions(-)
Building cmakefiles and deps
Building main product

Testing

Started testing at 2021/12/04 11:31:27
Testing mode: rebalance,n1ql,functional
Starting server
Starting server: attempt 1

Functional tests

2021/12/04 11:33:08 Started functional
Post "http://127.0.0.1:9000/pools/default": EOF
Quota setup Post "http://127.0.0.1:9000/pools/default": EOF
2021/12/04 11:33:19 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 11:33:25 Error in adding nodes : , response: {"otpNode":"n_1@127.0.0.1"}
2021/12/04 11:33:25 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 11:33:30 Removing node(s): [] from the cluster
2021/12/04 11:33:35 Rebalance progress: 2.69458912037037
2021/12/04 11:33:40 Rebalance progress: 7.288049768518519
2021/12/04 11:33:45 Rebalance progress: 10.52517361111111
2021/12/04 11:33:50 Rebalance progress: 15.0462962962963
2021/12/04 11:33:55 Rebalance progress: 18.51851851851852
2021/12/04 11:34:00 Rebalance progress: 22.96730324074074
2021/12/04 11:34:05 Rebalance progress: 27.54267939814815
2021/12/04 11:34:10 Rebalance progress: 100
=== RUN   TestAdvancedGetOps
2021/12/04 11:34:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:34:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:34:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:35:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:35:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:35:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:35:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:36:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:36:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:36:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:36:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:37:01 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:37:16 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:37:31 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:37:46 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:hello-world Completion:0}]
2021/12/04 11:38:01 Deploying app: TestAdvancedGetOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:38:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedGetOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:38:07 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 11:38:12 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 11:38:17 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 11:38:22 Waiting for app: TestAdvancedGetOps to get deployed
2021/12/04 11:38:22 App: TestAdvancedGetOps got deployed
2021/12/04 11:38:22 Starting bucket ops to source bucket
2021/12/04 11:38:27 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:38:32 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:38:32 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 11:38:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:38:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:38:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:38:37 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 11:38:42 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 11:38:47 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 11:38:52 Waiting for app: TestAdvancedGetOps to get un-deployed
2021/12/04 11:38:52 App: TestAdvancedGetOps got un-deployed
2021/12/04 11:38:52 No eventing-con process running
2021/12/04 11:38:52 Function: TestAdvancedGetOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedGetOps deleting in the background"
}
2021/12/04 11:38:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:39:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:39:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:39:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:39:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:39:21 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/04 11:39:26 Waiting for dst bucket item count to get to: 0 curr count: 99
2021/12/04 11:39:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedGetOps (310.27s)
=== RUN   TestAdvancedInsertOps
2021/12/04 11:39:26 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:39:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:39:32 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:39:37 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:39:42 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:39:47 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:39:47 App: TestAdvancedInsertOps got deployed
2021/12/04 11:39:47 Starting bucket ops to source bucket
2021/12/04 11:39:53 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 11:39:58 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/04 11:40:03 Waiting for dst bucket item count to get to: 1024 curr count: 512
2021/12/04 11:40:03 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 11:40:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:40:03 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:40:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:40:08 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:40:13 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:40:18 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:40:23 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:40:23 App: TestAdvancedInsertOps got un-deployed
2021/12/04 11:40:23 No eventing-con process running
2021/12/04 11:40:23 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/04 11:40:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:40:35 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:40:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:40:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:40:47 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:40:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:40:47 Testing insert operation on source bucket
2021/12/04 11:40:47 Deploying app: TestAdvancedInsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:40:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:40:53 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:40:58 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:41:03 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:41:08 Waiting for app: TestAdvancedInsertOps to get deployed
2021/12/04 11:41:08 App: TestAdvancedInsertOps got deployed
2021/12/04 11:41:08 Starting bucket ops to source bucket
2021/12/04 11:41:14 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 11:41:19 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 11:41:19 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 11:41:19 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:41:19 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:41:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:41:24 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:41:29 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:41:34 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:41:39 Waiting for app: TestAdvancedInsertOps to get un-deployed
2021/12/04 11:41:39 App: TestAdvancedInsertOps got un-deployed
2021/12/04 11:41:39 No eventing-con process running
2021/12/04 11:41:39 Function: TestAdvancedInsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertOps deleting in the background"
}
2021/12/04 11:41:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:41:51 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 11:41:56 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 11:41:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:41:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:42:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertOps (156.74s)
=== RUN   TestAdvancedUpsertOps
2021/12/04 11:42:03 Starting bucket ops to source bucket
2021/12/04 11:42:03 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:42:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:42:10 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:42:15 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:42:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:42:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:42:25 App: TestAdvancedUpsertOps got deployed
2021/12/04 11:42:25 Starting bucket ops to source bucket
2021/12/04 11:42:31 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:42:36 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/04 11:42:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:42:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:42:36 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:42:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:42:41 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:42:46 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:42:51 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:42:56 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:42:56 App: TestAdvancedUpsertOps got un-deployed
2021/12/04 11:42:56 No eventing-con process running
2021/12/04 11:42:56 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/04 11:42:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:43:07 Waiting for dst bucket item count to get to: 0 curr count: 512
2021/12/04 11:43:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:43:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:43:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:43:14 Testing upsert operation on source bucket
2021/12/04 11:43:14 Deploying app: TestAdvancedUpsertOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:43:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedUpsertOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:43:20 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:43:25 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:43:30 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:43:36 Waiting for app: TestAdvancedUpsertOps to get deployed
2021/12/04 11:43:36 App: TestAdvancedUpsertOps got deployed
2021/12/04 11:43:36 Starting bucket ops to source bucket
2021/12/04 11:43:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:43:37 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:43:37 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:43:37 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:43:42 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:43:47 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:43:52 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:43:57 Waiting for app: TestAdvancedUpsertOps to get un-deployed
2021/12/04 11:43:57 App: TestAdvancedUpsertOps got un-deployed
2021/12/04 11:43:57 No eventing-con process running
2021/12/04 11:43:57 Function: TestAdvancedUpsertOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedUpsertOps deleting in the background"
}
2021/12/04 11:43:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:44:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:44:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:44:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedUpsertOps (127.95s)
=== RUN   TestAdvancedReplaceOps
2021/12/04 11:44:11 Starting bucket ops to source bucket
2021/12/04 11:44:12 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:44:13 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:44:18 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:44:23 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:44:28 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:44:33 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:44:33 App: TestAdvancedReplaceOps got deployed
2021/12/04 11:44:33 Starting bucket ops to source bucket
2021/12/04 11:44:39 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:44:44 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:44:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:44:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:44:44 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:44:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:44:49 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:44:54 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:44:59 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:45:04 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:45:04 App: TestAdvancedReplaceOps got un-deployed
2021/12/04 11:45:04 No eventing-con process running
2021/12/04 11:45:04 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/04 11:45:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:45:16 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 11:45:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:45:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:45:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:45:23 Testing upsert operation on source bucket
2021/12/04 11:45:23 Deploying app: TestAdvancedReplaceOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:45:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedReplaceOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:45:29 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:45:34 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:45:39 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:45:44 Waiting for app: TestAdvancedReplaceOps to get deployed
2021/12/04 11:45:44 App: TestAdvancedReplaceOps got deployed
2021/12/04 11:45:44 Starting bucket ops to source bucket
2021/12/04 11:45:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:45:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:45:45 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:45:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:45:50 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:45:55 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:46:00 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:46:05 Waiting for app: TestAdvancedReplaceOps to get un-deployed
2021/12/04 11:46:05 App: TestAdvancedReplaceOps got un-deployed
2021/12/04 11:46:05 No eventing-con process running
2021/12/04 11:46:05 Function: TestAdvancedReplaceOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedReplaceOps deleting in the background"
}
2021/12/04 11:46:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:46:17 Waiting for dst bucket item count to get to: 0 curr count: 400
2021/12/04 11:46:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:46:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:46:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedReplaceOps (133.00s)
=== RUN   TestAdvancedDeleteOps
2021/12/04 11:46:24 Starting bucket ops to source bucket
2021/12/04 11:46:24 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:46:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:46:30 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:46:35 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:46:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:46:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:46:45 App: TestAdvancedDeleteOps got deployed
2021/12/04 11:46:45 Starting bucket ops to source bucket
2021/12/04 11:46:50 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:46:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:46:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:46:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:46:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:46:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:47:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:47:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:47:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:47:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:47:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/04 11:47:15 No eventing-con process running
2021/12/04 11:47:15 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/04 11:47:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:47:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:47:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:47:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:47:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:47:34 Testing delete operation on source bucket
2021/12/04 11:47:34 Deploying app: TestAdvancedDeleteOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:47:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedDeleteOps' in metakv",
  "warnings": null
 }
}
2021/12/04 11:47:40 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:47:45 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:47:50 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:47:55 Waiting for app: TestAdvancedDeleteOps to get deployed
2021/12/04 11:47:55 App: TestAdvancedDeleteOps got deployed
2021/12/04 11:47:55 Starting bucket ops to source bucket
2021/12/04 11:47:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:47:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:47:55 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:47:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:48:00 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:48:05 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:48:10 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:48:15 Waiting for app: TestAdvancedDeleteOps to get un-deployed
2021/12/04 11:48:15 App: TestAdvancedDeleteOps got un-deployed
2021/12/04 11:48:15 No eventing-con process running
2021/12/04 11:48:16 Function: TestAdvancedDeleteOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedDeleteOps deleting in the background"
}
2021/12/04 11:48:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:48:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:48:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:48:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedDeleteOps (125.88s)
=== RUN   TestEnoentAdvancedGet
2021/12/04 11:48:30 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:48:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/04 11:48:36 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:48:41 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:48:46 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:48:51 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:48:51 App: TestEnoentAdvancedGet got deployed
2021/12/04 11:48:51 Starting bucket ops to source bucket
2021/12/04 11:48:56 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:49:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:49:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 11:49:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:49:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:49:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:49:06 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:49:11 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:49:16 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:49:21 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:49:21 App: TestEnoentAdvancedGet got un-deployed
2021/12/04 11:49:21 No eventing-con process running
2021/12/04 11:49:21 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/04 11:49:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:49:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:49:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:49:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:49:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:49:50 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:49:55 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:49:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:49:55 Testing get enoent operation on source bucket
2021/12/04 11:49:55 Deploying app: TestEnoentAdvancedGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:49:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedGet' in metakv",
  "warnings": null
 }
}
2021/12/04 11:50:01 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:50:06 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:50:11 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:50:16 Waiting for app: TestEnoentAdvancedGet to get deployed
2021/12/04 11:50:16 App: TestEnoentAdvancedGet got deployed
2021/12/04 11:50:16 Starting bucket ops to source bucket
2021/12/04 11:50:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 11:50:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 11:50:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:50:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:50:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:50:26 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:50:31 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:50:36 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:50:41 Waiting for app: TestEnoentAdvancedGet to get un-deployed
2021/12/04 11:50:41 App: TestEnoentAdvancedGet got un-deployed
2021/12/04 11:50:41 No eventing-con process running
2021/12/04 11:50:41 Function: TestEnoentAdvancedGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedGet deleting in the background"
}
2021/12/04 11:50:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:50:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/04 11:50:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:50:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:51:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:51:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedGet (155.14s)
=== RUN   TestEnoentAdvancedReplace
2021/12/04 11:51:05 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:51:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 11:51:11 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:51:16 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:51:21 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:51:26 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:51:26 App: TestEnoentAdvancedReplace got deployed
2021/12/04 11:51:26 Starting bucket ops to source bucket
2021/12/04 11:51:31 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:51:31 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 11:51:31 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:51:31 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:51:31 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:51:36 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:51:41 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:51:46 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:51:51 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:51:51 App: TestEnoentAdvancedReplace got un-deployed
2021/12/04 11:51:51 No eventing-con process running
2021/12/04 11:51:51 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/04 11:51:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:52:03 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:52:08 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:52:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:52:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:52:20 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:52:25 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:52:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:52:25 Testing get enoent operation on source bucket
2021/12/04 11:52:25 Deploying app: TestEnoentAdvancedReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:52:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 11:52:31 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:52:36 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:52:41 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:52:46 Waiting for app: TestEnoentAdvancedReplace to get deployed
2021/12/04 11:52:46 App: TestEnoentAdvancedReplace got deployed
2021/12/04 11:52:46 Starting bucket ops to source bucket
2021/12/04 11:52:51 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 11:52:51 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 11:52:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:52:51 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:52:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:52:56 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:53:01 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:53:06 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:53:11 Waiting for app: TestEnoentAdvancedReplace to get un-deployed
2021/12/04 11:53:11 App: TestEnoentAdvancedReplace got un-deployed
2021/12/04 11:53:11 No eventing-con process running
2021/12/04 11:53:11 Function: TestEnoentAdvancedReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedReplace deleting in the background"
}
2021/12/04 11:53:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:53:23 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/04 11:53:28 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:53:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:53:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:53:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedReplace (150.17s)
=== RUN   TestEnoentAdvancedDelete
2021/12/04 11:53:35 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:53:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 11:53:41 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:53:46 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:53:51 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:53:56 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:53:56 App: TestEnoentAdvancedDelete got deployed
2021/12/04 11:53:56 Starting bucket ops to source bucket
2021/12/04 11:54:01 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 11:54:01 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 11:54:01 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:54:01 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:54:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:54:06 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:54:11 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:54:16 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:54:21 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:54:21 App: TestEnoentAdvancedDelete got un-deployed
2021/12/04 11:54:21 No eventing-con process running
2021/12/04 11:54:21 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/04 11:54:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:54:33 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:54:38 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:54:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:54:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:54:50 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/04 11:54:55 Waiting for dst bucket item count to get to: 0 curr count: 59
2021/12/04 11:54:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:54:55 Testing get enoent operation on source bucket
2021/12/04 11:54:55 Deploying app: TestEnoentAdvancedDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:54:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentAdvancedDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 11:55:01 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:55:06 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:55:11 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:55:16 Waiting for app: TestEnoentAdvancedDelete to get deployed
2021/12/04 11:55:16 App: TestEnoentAdvancedDelete got deployed
2021/12/04 11:55:16 Starting bucket ops to source bucket
2021/12/04 11:55:21 Waiting for dst bucket item count to get to: 200 curr count: 0
2021/12/04 11:55:21 src & dst bucket item count matched up. src bucket count: 200 dst bucket count: 200
2021/12/04 11:55:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:55:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:55:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:55:26 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:55:31 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:55:36 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:55:41 Waiting for app: TestEnoentAdvancedDelete to get un-deployed
2021/12/04 11:55:41 App: TestEnoentAdvancedDelete got un-deployed
2021/12/04 11:55:41 No eventing-con process running
2021/12/04 11:55:41 Function: TestEnoentAdvancedDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentAdvancedDelete deleting in the background"
}
2021/12/04 11:55:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:55:53 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/04 11:55:58 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:55:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:56:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:56:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentAdvancedDelete (150.07s)
=== RUN   TestAdvancedInsertKeyExist
2021/12/04 11:56:05 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:56:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/04 11:56:11 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:56:16 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:56:21 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:56:26 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:56:26 App: TestAdvancedInsertKeyExist got deployed
2021/12/04 11:56:26 Starting bucket ops to source bucket
2021/12/04 11:56:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:56:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:56:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:56:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:56:31 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:56:36 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:56:41 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:56:46 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:56:46 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/04 11:56:46 No eventing-con process running
2021/12/04 11:56:46 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/04 11:56:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:56:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:56:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:57:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:57:01 Pumping document in the destination bucket
2021/12/04 11:57:01 Starting bucket ops to source bucket
2021/12/04 11:57:01 Deploying app: TestAdvancedInsertKeyExist. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:57:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAdvancedInsertKeyExist' in metakv",
  "warnings": null
 }
}
2021/12/04 11:57:07 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:57:12 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:57:17 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:57:22 Waiting for app: TestAdvancedInsertKeyExist to get deployed
2021/12/04 11:57:22 App: TestAdvancedInsertKeyExist got deployed
2021/12/04 11:57:22 Starting bucket ops to source bucket
2021/12/04 11:57:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:57:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:57:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:57:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:57:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:57:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:57:37 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:57:42 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:57:47 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:57:52 Waiting for app: TestAdvancedInsertKeyExist to get un-deployed
2021/12/04 11:57:52 App: TestAdvancedInsertKeyExist got un-deployed
2021/12/04 11:57:52 No eventing-con process running
2021/12/04 11:57:52 Function: TestAdvancedInsertKeyExist delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAdvancedInsertKeyExist deleting in the background"
}
2021/12/04 11:57:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:58:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:58:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:58:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:58:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:58:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAdvancedInsertKeyExist (130.77s)
=== RUN   TestExipryGet
2021/12/04 11:58:16 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 11:58:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/04 11:58:22 Waiting for app: TestExipryGet to get deployed
2021/12/04 11:58:27 Waiting for app: TestExipryGet to get deployed
2021/12/04 11:58:32 Waiting for app: TestExipryGet to get deployed
2021/12/04 11:58:37 Waiting for app: TestExipryGet to get deployed
2021/12/04 11:58:37 App: TestExipryGet got deployed
2021/12/04 11:58:37 Starting bucket ops to source bucket
2021/12/04 11:59:22 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 11:59:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:59:32 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 11:59:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 11:59:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 11:59:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 11:59:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 11:59:37 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 11:59:42 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 11:59:47 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 11:59:52 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 11:59:52 App: TestExipryGet got un-deployed
2021/12/04 11:59:52 No eventing-con process running
2021/12/04 11:59:52 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/04 11:59:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 11:59:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:00:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:00:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:00:06 Pumping document in the destination bucket for expiry get
2021/12/04 12:00:06 Starting bucket ops to source bucket
2021/12/04 12:00:07 Deploying app: TestExipryGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:00:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryGet' in metakv",
  "warnings": null
 }
}
2021/12/04 12:00:13 Waiting for app: TestExipryGet to get deployed
2021/12/04 12:00:18 Waiting for app: TestExipryGet to get deployed
2021/12/04 12:00:23 Waiting for app: TestExipryGet to get deployed
2021/12/04 12:00:28 Waiting for app: TestExipryGet to get deployed
2021/12/04 12:00:28 App: TestExipryGet got deployed
2021/12/04 12:00:28 Starting bucket ops to source bucket
2021/12/04 12:01:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:01:18 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:01:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:01:18 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:01:18 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:01:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:01:23 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 12:01:28 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 12:01:33 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 12:01:38 Waiting for app: TestExipryGet to get un-deployed
2021/12/04 12:01:38 App: TestExipryGet got un-deployed
2021/12/04 12:01:38 No eventing-con process running
2021/12/04 12:01:38 Function: TestExipryGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryGet deleting in the background"
}
2021/12/04 12:01:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:01:50 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/04 12:01:55 Waiting for dst bucket item count to get to: 0 curr count: 57
2021/12/04 12:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:01:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryGet (226.04s)
=== RUN   TestExipryInsert
2021/12/04 12:02:02 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:02:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/04 12:02:08 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:02:13 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:02:18 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:02:23 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:02:23 App: TestExipryInsert got deployed
2021/12/04 12:02:23 Starting bucket ops to source bucket
2021/12/04 12:03:08 Waiting for dst bucket item count to get to: 0 curr count: 200
2021/12/04 12:03:13 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:03:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:03:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:03:13 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:03:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:03:18 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:03:23 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:03:28 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:03:33 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:03:33 App: TestExipryInsert got un-deployed
2021/12/04 12:03:33 No eventing-con process running
2021/12/04 12:03:33 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/04 12:03:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:03:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:03:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:03:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:03:48 Testing on the destination bucket for expiry insert
2021/12/04 12:03:48 Deploying app: TestExipryInsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:03:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryInsert' in metakv",
  "warnings": null
 }
}
2021/12/04 12:03:54 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:03:59 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:04:04 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:04:09 Waiting for app: TestExipryInsert to get deployed
2021/12/04 12:04:09 App: TestExipryInsert got deployed
2021/12/04 12:04:09 Starting bucket ops to source bucket
2021/12/04 12:04:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:04:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:04:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:04:59 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:04:59 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:04:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:05:04 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:05:09 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:05:14 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:05:19 Waiting for app: TestExipryInsert to get un-deployed
2021/12/04 12:05:19 App: TestExipryInsert got un-deployed
2021/12/04 12:05:19 No eventing-con process running
2021/12/04 12:05:19 Function: TestExipryInsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryInsert deleting in the background"
}
2021/12/04 12:05:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:05:31 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/04 12:05:36 Waiting for dst bucket item count to get to: 0 curr count: 96
2021/12/04 12:05:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:05:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:05:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryInsert (220.93s)
=== RUN   TestExipryReplace
2021/12/04 12:05:43 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:05:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 12:05:49 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:05:54 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:05:59 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:06:04 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:06:04 App: TestExipryReplace got deployed
2021/12/04 12:06:04 Starting bucket ops to source bucket
2021/12/04 12:06:49 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:06:54 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:06:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:06:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:06:54 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:06:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:06:59 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:07:04 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:07:09 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:07:14 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:07:14 App: TestExipryReplace got un-deployed
2021/12/04 12:07:14 No eventing-con process running
2021/12/04 12:07:14 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/04 12:07:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:07:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:07:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:07:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:07:28 Testing on the destination bucket for expiry insert
2021/12/04 12:07:28 Deploying app: TestExipryReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:07:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 12:07:34 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:07:40 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:07:45 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:07:50 Waiting for app: TestExipryReplace to get deployed
2021/12/04 12:07:50 App: TestExipryReplace got deployed
2021/12/04 12:07:50 Starting bucket ops to source bucket
2021/12/04 12:08:35 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:08:40 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:08:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:08:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:08:40 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:08:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:08:45 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:08:50 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:08:55 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:09:00 Waiting for app: TestExipryReplace to get un-deployed
2021/12/04 12:09:00 App: TestExipryReplace got un-deployed
2021/12/04 12:09:00 No eventing-con process running
2021/12/04 12:09:00 Function: TestExipryReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryReplace deleting in the background"
}
2021/12/04 12:09:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:09:12 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:09:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:09:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:09:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:09:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryReplace (220.78s)
=== RUN   TestExipryUpsert
2021/12/04 12:09:23 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:09:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/04 12:09:30 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:09:35 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:09:40 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:09:45 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:09:45 App: TestExipryUpsert got deployed
2021/12/04 12:09:45 Starting bucket ops to source bucket
2021/12/04 12:10:30 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:10:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:10:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:10:30 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:10:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:10:35 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:10:40 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:10:45 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:10:50 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:10:50 App: TestExipryUpsert got un-deployed
2021/12/04 12:10:50 No eventing-con process running
2021/12/04 12:10:50 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/04 12:10:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:10:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:10:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:11:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:11:04 Testing on the destination bucket for expiry insert
2021/12/04 12:11:04 Deploying app: TestExipryUpsert. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:11:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestExipryUpsert' in metakv",
  "warnings": null
 }
}
2021/12/04 12:11:10 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:11:15 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:11:20 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:11:25 Waiting for app: TestExipryUpsert to get deployed
2021/12/04 12:11:25 App: TestExipryUpsert got deployed
2021/12/04 12:11:25 Starting bucket ops to source bucket
2021/12/04 12:12:11 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:12:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:12:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:12:11 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:12:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:12:16 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:12:21 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:12:26 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:12:31 Waiting for app: TestExipryUpsert to get un-deployed
2021/12/04 12:12:31 App: TestExipryUpsert got un-deployed
2021/12/04 12:12:31 No eventing-con process running
2021/12/04 12:12:31 Function: TestExipryUpsert delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestExipryUpsert deleting in the background"
}
2021/12/04 12:12:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:12:42 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:12:47 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:12:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:12:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:12:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestExipryUpsert (210.78s)
=== RUN   TestCasReplace
2021/12/04 12:12:54 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:12:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 12:13:01 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:13:06 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:13:11 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:13:16 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:13:16 App: TestCasReplace got deployed
2021/12/04 12:13:16 Starting bucket ops to source bucket
2021/12/04 12:13:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:13:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:13:16 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:13:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:13:21 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:13:26 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:13:31 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:13:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:13:36 App: TestCasReplace got un-deployed
2021/12/04 12:13:36 No eventing-con process running
2021/12/04 12:13:36 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/04 12:13:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:13:48 Waiting for dst bucket item count to get to: 0 curr count: 2
2021/12/04 12:13:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:13:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:13:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:13:55 Testing on the destination bucket for upsert with cas
2021/12/04 12:13:55 Starting bucket ops to source bucket
2021/12/04 12:13:55 Deploying app: TestCasReplace. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:13:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasReplace' in metakv",
  "warnings": null
 }
}
2021/12/04 12:14:01 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:14:06 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:14:11 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:14:16 Waiting for app: TestCasReplace to get deployed
2021/12/04 12:14:16 App: TestCasReplace got deployed
2021/12/04 12:14:16 Starting bucket ops to source bucket
2021/12/04 12:14:21 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:14:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:14:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:14:21 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:14:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:14:26 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:14:31 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:14:36 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:14:41 Waiting for app: TestCasReplace to get un-deployed
2021/12/04 12:14:41 App: TestCasReplace got un-deployed
2021/12/04 12:14:41 No eventing-con process running
2021/12/04 12:14:41 Function: TestCasReplace delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasReplace deleting in the background"
}
2021/12/04 12:14:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:14:53 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:14:58 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:14:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:15:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:15:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasReplace (130.47s)
=== RUN   TestCasDelete
2021/12/04 12:15:05 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:15:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 12:15:11 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:15:16 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:15:21 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:15:26 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:15:26 App: TestCasDelete got deployed
2021/12/04 12:15:26 Starting bucket ops to source bucket
2021/12/04 12:15:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:15:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:15:26 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:15:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:15:31 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:15:36 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:15:41 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:15:46 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:15:46 App: TestCasDelete got un-deployed
2021/12/04 12:15:46 No eventing-con process running
2021/12/04 12:15:46 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/04 12:15:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:15:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:15:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:16:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:16:00 Testing on the destination bucket for delete with cas
2021/12/04 12:16:00 Starting bucket ops to source bucket
2021/12/04 12:16:01 Deploying app: TestCasDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:16:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCasDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 12:16:07 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:16:12 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:16:17 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:16:22 Waiting for app: TestCasDelete to get deployed
2021/12/04 12:16:22 App: TestCasDelete got deployed
2021/12/04 12:16:22 Starting bucket ops to source bucket
2021/12/04 12:16:27 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:16:32 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:16:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:16:32 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:16:32 http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 12:16:32 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:16:37 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:16:42 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:16:47 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:16:52 Waiting for app: TestCasDelete to get un-deployed
2021/12/04 12:16:52 App: TestCasDelete got un-deployed
2021/12/04 12:16:52 No eventing-con process running
2021/12/04 12:16:52 Function: TestCasDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCasDelete deleting in the background"
}
2021/12/04 12:16:54 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:17:04 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:17:09 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:17:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:17:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:17:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCasDelete (131.01s)
=== RUN   TestCountersIncrement
2021/12/04 12:17:16 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 12:17:22 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 12:17:22 Removing node(s): [] from the cluster
2021/12/04 12:17:27 Rebalance progress: 100
2021/12/04 12:17:27 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:17:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/04 12:17:33 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:17:38 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:17:43 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:17:48 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:17:48 App: TestCountersIncrement got deployed
2021/12/04 12:17:48 Starting bucket ops to source bucket
2021/12/04 12:17:54 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 12:17:59 Waiting for dst bucket item count to get to: 1024 curr count: 463
2021/12/04 12:18:04 Waiting for dst bucket item count to get to: 1024 curr count: 463
2021/12/04 12:18:04 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 12:18:04 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:18:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:18:10 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:18:15 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:18:20 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:18:25 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:18:25 App: TestCountersIncrement got un-deployed
2021/12/04 12:18:25 No eventing-con process running
2021/12/04 12:18:25 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/04 12:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:18:36 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/04 12:18:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:18:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:18:48 Waiting for dst bucket item count to get to: 0 curr count: 803
2021/12/04 12:18:53 Waiting for dst bucket item count to get to: 0 curr count: 291
2021/12/04 12:18:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:18:53 Deploying app: TestCountersIncrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:18:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersIncrement' in metakv",
  "warnings": null
 }
}
2021/12/04 12:18:59 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:19:04 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:19:09 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:19:14 Waiting for app: TestCountersIncrement to get deployed
2021/12/04 12:19:14 App: TestCountersIncrement got deployed
2021/12/04 12:19:14 Starting bucket ops to source bucket
2021/12/04 12:19:21 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 12:19:26 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 12:19:26 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 12:19:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:19:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:19:31 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:19:36 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:19:41 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:19:46 Waiting for app: TestCountersIncrement to get un-deployed
2021/12/04 12:19:46 App: TestCountersIncrement got un-deployed
2021/12/04 12:19:46 No eventing-con process running
2021/12/04 12:19:46 Function: TestCountersIncrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersIncrement deleting in the background"
}
2021/12/04 12:19:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:19:58 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 12:19:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:19:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:20:09 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 12:20:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:20:09 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/04 12:20:14 Rebalance progress: 100
--- PASS: TestCountersIncrement (178.65s)
=== RUN   TestCountersDecrement
2021/12/04 12:20:14 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 12:20:15 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 12:20:15 Removing node(s): [] from the cluster
2021/12/04 12:20:20 Rebalance progress: 100
2021/12/04 12:20:20 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:20:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/04 12:20:26 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:20:31 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:20:36 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:20:41 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:20:41 App: TestCountersDecrement got deployed
2021/12/04 12:20:41 Starting bucket ops to source bucket
2021/12/04 12:20:47 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 12:20:52 Waiting for dst bucket item count to get to: 1024 curr count: 0
2021/12/04 12:20:52 src & dst bucket item count matched up. src bucket count: 1024 dst bucket count: 1024
2021/12/04 12:20:52 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:20:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:20:57 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:21:02 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:21:08 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:21:13 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:21:13 App: TestCountersDecrement got un-deployed
2021/12/04 12:21:13 No eventing-con process running
2021/12/04 12:21:13 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/04 12:21:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:21:24 Waiting for dst bucket item count to get to: 0 curr count: 1025
2021/12/04 12:21:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:21:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:21:36 Waiting for dst bucket item count to get to: 0 curr count: 1024
2021/12/04 12:21:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:21:36 Deploying app: TestCountersDecrement. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:21:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCountersDecrement' in metakv",
  "warnings": null
 }
}
2021/12/04 12:21:42 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:21:47 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:21:52 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:21:57 Waiting for app: TestCountersDecrement to get deployed
2021/12/04 12:21:57 App: TestCountersDecrement got deployed
2021/12/04 12:21:57 Starting bucket ops to source bucket
2021/12/04 12:22:04 Waiting for dst bucket item count to get to: 2048 curr count: 0
2021/12/04 12:22:09 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 12:22:14 Waiting for dst bucket item count to get to: 2048 curr count: 1024
2021/12/04 12:22:14 src & dst bucket item count matched up. src bucket count: 2048 dst bucket count: 2048
2021/12/04 12:22:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:22:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:22:19 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:22:24 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:22:29 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:22:34 Waiting for app: TestCountersDecrement to get un-deployed
2021/12/04 12:22:34 App: TestCountersDecrement got un-deployed
2021/12/04 12:22:34 No eventing-con process running
2021/12/04 12:22:34 Function: TestCountersDecrement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCountersDecrement deleting in the background"
}
2021/12/04 12:22:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:22:46 Waiting for dst bucket item count to get to: 0 curr count: 2048
2021/12/04 12:22:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:22:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:22:57 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 12:22:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:22:57 Removing node(s): [http://127.0.0.1:19002 http://127.0.0.1:19003] from the cluster
2021/12/04 12:23:02 Rebalance progress: 100
--- PASS: TestCountersDecrement (168.03s)
=== RUN   TestCacheGet
2021/12/04 12:23:02 Deploying app: TestCacheGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:23:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheGet' in metakv",
  "warnings": null
 }
}
2021/12/04 12:23:09 Waiting for app: TestCacheGet to get deployed
2021/12/04 12:23:14 Waiting for app: TestCacheGet to get deployed
2021/12/04 12:23:19 Waiting for app: TestCacheGet to get deployed
2021/12/04 12:23:24 Waiting for app: TestCacheGet to get deployed
2021/12/04 12:23:24 App: TestCacheGet got deployed
2021/12/04 12:23:24 Starting bucket ops to source bucket
2021/12/04 12:23:29 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/04 12:23:34 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/04 12:23:39 Waiting for dst bucket item count to get to: 20 curr count: 0
2021/12/04 12:23:44 Waiting for dst bucket item count to get to: 20 curr count: 8
2021/12/04 12:23:49 Waiting for dst bucket item count to get to: 20 curr count: 16
2021/12/04 12:23:54 Waiting for dst bucket item count to get to: 20 curr count: 18
2021/12/04 12:23:54 src & dst bucket item count matched up. src bucket count: 20 dst bucket count: 20
2021/12/04 12:23:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:23:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:23:59 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 12:24:04 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 12:24:09 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 12:24:14 Waiting for app: TestCacheGet to get un-deployed
2021/12/04 12:24:14 App: TestCacheGet got un-deployed
2021/12/04 12:24:14 No eventing-con process running
2021/12/04 12:24:14 Function: TestCacheGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheGet deleting in the background"
}
2021/12/04 12:24:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:24:26 Waiting for dst bucket item count to get to: 0 curr count: 20
2021/12/04 12:24:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:24:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:24:38 Waiting for dst bucket item count to get to: 0 curr count: 10
2021/12/04 12:24:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheGet (95.33s)
=== RUN   TestCacheRYOW
2021/12/04 12:24:38 Deploying app: TestCacheRYOW. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:24:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheRYOW' in metakv",
  "warnings": null
 }
}
2021/12/04 12:24:44 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 12:24:49 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 12:24:54 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 12:24:59 Waiting for app: TestCacheRYOW to get deployed
2021/12/04 12:24:59 App: TestCacheRYOW got deployed
2021/12/04 12:24:59 Starting bucket ops to source bucket
2021/12/04 12:25:04 Waiting for dst bucket item count to get to: 50 curr count: 0
2021/12/04 12:25:09 Waiting for dst bucket item count to get to: 50 curr count: 25
2021/12/04 12:25:09 src & dst bucket item count matched up. src bucket count: 50 dst bucket count: 50
2021/12/04 12:25:09 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:25:09 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:25:14 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 12:25:19 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 12:25:24 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 12:25:29 Waiting for app: TestCacheRYOW to get un-deployed
2021/12/04 12:25:29 App: TestCacheRYOW got un-deployed
2021/12/04 12:25:29 No eventing-con process running
2021/12/04 12:25:30 Function: TestCacheRYOW delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheRYOW deleting in the background"
}
2021/12/04 12:25:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:25:41 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:25:46 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:25:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:25:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:25:58 Waiting for dst bucket item count to get to: 0 curr count: 37
2021/12/04 12:26:03 Waiting for dst bucket item count to get to: 0 curr count: 12
2021/12/04 12:26:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheRYOW (85.41s)
=== RUN   TestCacheExpiry
2021/12/04 12:26:03 Deploying app: TestCacheExpiry. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:26:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiry' in metakv",
  "warnings": null
 }
}
2021/12/04 12:26:09 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 12:26:14 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 12:26:19 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 12:26:24 Waiting for app: TestCacheExpiry to get deployed
2021/12/04 12:26:24 App: TestCacheExpiry got deployed
2021/12/04 12:26:24 Starting bucket ops to source bucket
2021/12/04 12:26:30 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:26:35 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/04 12:26:40 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/04 12:26:45 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/04 12:26:50 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/04 12:26:50 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:26:50 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:26:50 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:26:55 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 12:27:00 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 12:27:05 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 12:27:10 Waiting for app: TestCacheExpiry to get un-deployed
2021/12/04 12:27:10 App: TestCacheExpiry got un-deployed
2021/12/04 12:27:10 No eventing-con process running
2021/12/04 12:27:10 Function: TestCacheExpiry delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiry deleting in the background"
}
2021/12/04 12:27:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:27:22 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:27:27 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:27:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:27:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:27:38 Waiting for dst bucket item count to get to: 0 curr count: 81
2021/12/04 12:27:43 Waiting for dst bucket item count to get to: 0 curr count: 31
2021/12/04 12:27:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiry (100.23s)
=== RUN   TestCacheExpiryCustom
2021/12/04 12:27:43 Deploying app: TestCacheExpiryCustom. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:27:45 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheExpiryCustom' in metakv",
  "warnings": null
 }
}
2021/12/04 12:27:50 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 12:27:55 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 12:28:00 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 12:28:05 Waiting for app: TestCacheExpiryCustom to get deployed
2021/12/04 12:28:05 App: TestCacheExpiryCustom got deployed
2021/12/04 12:28:05 Starting bucket ops to source bucket
2021/12/04 12:28:10 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:28:15 Waiting for dst bucket item count to get to: 100 curr count: 14
2021/12/04 12:28:20 Waiting for dst bucket item count to get to: 100 curr count: 37
2021/12/04 12:28:25 Waiting for dst bucket item count to get to: 100 curr count: 72
2021/12/04 12:28:30 Waiting for dst bucket item count to get to: 100 curr count: 99
2021/12/04 12:28:30 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:28:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:28:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:28:35 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 12:28:40 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 12:28:45 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 12:28:50 Waiting for app: TestCacheExpiryCustom to get un-deployed
2021/12/04 12:28:50 App: TestCacheExpiryCustom got un-deployed
2021/12/04 12:28:50 No eventing-con process running
2021/12/04 12:28:50 Function: TestCacheExpiryCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheExpiryCustom deleting in the background"
}
2021/12/04 12:28:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:29:02 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:29:07 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:29:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:29:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:29:19 Waiting for dst bucket item count to get to: 0 curr count: 91
2021/12/04 12:29:24 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/04 12:29:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheExpiryCustom (100.34s)
=== RUN   TestCacheOverflowCustom
2021/12/04 12:29:24 Deploying app: TestCacheOverflowCustom. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:29:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCacheOverflowCustom' in metakv",
  "warnings": null
 }
}
2021/12/04 12:29:30 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 12:29:35 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 12:29:40 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 12:29:45 Waiting for app: TestCacheOverflowCustom to get deployed
2021/12/04 12:29:45 App: TestCacheOverflowCustom got deployed
2021/12/04 12:29:45 Starting bucket ops to source bucket
2021/12/04 12:29:50 Waiting for dst bucket item count to get to: 35 curr count: 0
2021/12/04 12:29:55 Waiting for dst bucket item count to get to: 35 curr count: 5
2021/12/04 12:30:00 Waiting for dst bucket item count to get to: 35 curr count: 14
2021/12/04 12:30:05 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/04 12:30:10 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/04 12:30:15 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/04 12:30:20 Waiting for dst bucket item count to get to: 35 curr count: 27
2021/12/04 12:30:25 Waiting for dst bucket item count to get to: 35 curr count: 32
2021/12/04 12:30:30 Waiting for dst bucket item count to get to: 35 curr count: 32
2021/12/04 12:30:30 src & dst bucket item count matched up. src bucket count: 35 dst bucket count: 35
2021/12/04 12:30:30 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:30:30 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:30:35 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 12:30:40 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 12:30:45 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 12:30:50 Waiting for app: TestCacheOverflowCustom to get un-deployed
2021/12/04 12:30:50 App: TestCacheOverflowCustom got un-deployed
2021/12/04 12:30:50 No eventing-con process running
2021/12/04 12:30:50 Function: TestCacheOverflowCustom delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCacheOverflowCustom deleting in the background"
}
2021/12/04 12:30:52 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:31:02 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/04 12:31:07 Waiting for dst bucket item count to get to: 0 curr count: 5
2021/12/04 12:31:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:31:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:31:19 Waiting for dst bucket item count to get to: 0 curr count: 35
2021/12/04 12:31:24 Waiting for dst bucket item count to get to: 0 curr count: 17
2021/12/04 12:31:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCacheOverflowCustom (120.42s)
=== RUN   TestStrictMode
2021/12/04 12:31:24 Deploying app: TestStrictMode. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:31:25 Context: Post to main store, response code: 422 dump: {
 "name": "ERR_HANDLER_COMPILATION",
 "code": 27,
 "description": "Function compilation failed",
 "attributes": null,
 "runtime_info": {
  "code": 27,
  "info": {
   "area": "handlerCode",
   "column_number": 63,
   "compile_success": false,
   "description": "        destination[meta.id] = 00000000000000000000000000000101;\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nSyntaxError: Octal literals are not allowed in strict mode.\n",
   "index": 132,
   "language": "JavaScript",
   "line_number": 4
  }
 }
}
--- PASS: TestStrictMode (1.05s)
=== RUN   TestDataTypes
2021/12/04 12:31:25 Deploying app: TestDataTypes. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:31:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDataTypes' in metakv",
  "warnings": null
 }
}
2021/12/04 12:31:31 Waiting for app: TestDataTypes to get deployed
2021/12/04 12:31:36 Waiting for app: TestDataTypes to get deployed
2021/12/04 12:31:41 Waiting for app: TestDataTypes to get deployed
2021/12/04 12:31:46 Waiting for app: TestDataTypes to get deployed
2021/12/04 12:31:46 App: TestDataTypes got deployed
2021/12/04 12:31:46 Starting bucket ops to source bucket
2021/12/04 12:31:51 Waiting for dst bucket item count to get to: 18 curr count: 0
2021/12/04 12:31:51 src & dst bucket item count matched up. src bucket count: 18 dst bucket count: 18
2021/12/04 12:31:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:31:52 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:31:57 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 12:32:02 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 12:32:07 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 12:32:12 Waiting for app: TestDataTypes to get un-deployed
2021/12/04 12:32:12 App: TestDataTypes got un-deployed
2021/12/04 12:32:12 No eventing-con process running
2021/12/04 12:32:12 Function: TestDataTypes delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDataTypes deleting in the background"
}
2021/12/04 12:32:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:32:23 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 12:32:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:32:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:32:35 Waiting for dst bucket item count to get to: 0 curr count: 18
2021/12/04 12:32:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDataTypes (70.11s)
=== RUN   TestEnoentGet
2021/12/04 12:32:35 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:32:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/04 12:32:41 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:32:46 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:32:51 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:32:56 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:32:56 App: TestEnoentGet got deployed
2021/12/04 12:32:56 Starting bucket ops to source bucket
2021/12/04 12:33:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:33:02 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:33:02 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:33:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:33:07 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:33:12 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:33:17 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:33:22 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:33:22 App: TestEnoentGet got un-deployed
2021/12/04 12:33:22 No eventing-con process running
2021/12/04 12:33:22 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/04 12:33:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:33:34 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:33:39 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:33:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:33:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:33:51 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/04 12:33:56 Waiting for dst bucket item count to get to: 0 curr count: 80
2021/12/04 12:33:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:33:56 Deploying app: TestEnoentGet. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:33:57 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentGet' in metakv",
  "warnings": null
 }
}
2021/12/04 12:34:02 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:34:07 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:34:12 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:34:17 Waiting for app: TestEnoentGet to get deployed
2021/12/04 12:34:17 App: TestEnoentGet got deployed
2021/12/04 12:34:17 Starting bucket ops to source bucket
2021/12/04 12:34:22 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:34:22 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:34:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:34:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:34:27 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:34:32 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:34:37 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:34:42 Waiting for app: TestEnoentGet to get un-deployed
2021/12/04 12:34:42 App: TestEnoentGet got un-deployed
2021/12/04 12:34:42 No eventing-con process running
2021/12/04 12:34:42 Function: TestEnoentGet delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentGet deleting in the background"
}
2021/12/04 12:34:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:34:54 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:34:59 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:34:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:35:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:35:11 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/04 12:35:16 Waiting for dst bucket item count to get to: 0 curr count: 85
2021/12/04 12:35:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentGet (160.41s)
=== RUN   TestEnoentDelete
2021/12/04 12:35:16 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:35:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 12:35:22 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:35:27 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:35:32 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:35:37 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:35:37 App: TestEnoentDelete got deployed
2021/12/04 12:35:37 Starting bucket ops to source bucket
2021/12/04 12:35:42 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:35:42 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:35:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:35:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:35:47 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:35:52 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:35:57 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:36:02 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:36:02 App: TestEnoentDelete got un-deployed
2021/12/04 12:36:02 No eventing-con process running
2021/12/04 12:36:02 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/04 12:36:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:36:14 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:36:19 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:36:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:36:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:36:31 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/04 12:36:36 Waiting for dst bucket item count to get to: 0 curr count: 93
2021/12/04 12:36:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:36:36 Deploying app: TestEnoentDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:36:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEnoentDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 12:36:42 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:36:47 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:36:52 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:36:57 Waiting for app: TestEnoentDelete to get deployed
2021/12/04 12:36:57 App: TestEnoentDelete got deployed
2021/12/04 12:36:57 Starting bucket ops to source bucket
2021/12/04 12:37:02 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:37:07 Waiting for dst bucket item count to get to: 100 curr count: 78
2021/12/04 12:37:12 Waiting for dst bucket item count to get to: 100 curr count: 78
2021/12/04 12:37:12 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:37:12 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:37:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:37:17 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:37:22 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:37:27 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:37:32 Waiting for app: TestEnoentDelete to get un-deployed
2021/12/04 12:37:32 App: TestEnoentDelete got un-deployed
2021/12/04 12:37:32 No eventing-con process running
2021/12/04 12:37:33 Function: TestEnoentDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEnoentDelete deleting in the background"
}
2021/12/04 12:37:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:37:44 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:37:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:37:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:37:56 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEnoentDelete (160.46s)
=== RUN   TestError
2021/12/04 12:38:06 Waiting for app: TestError to get un-deployed
2021/12/04 12:38:06 App: TestError got un-deployed
2021/12/04 12:38:06 No eventing-con process running
2021/12/04 12:38:06 Function: TestError delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestError not found"
 }
}
2021/12/04 12:38:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:38:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:38:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:38:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:38:20 Deploying app: TestError. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:38:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestError' in metakv",
  "warnings": null
 }
}
2021/12/04 12:38:26 Waiting for app: TestError to get deployed
2021/12/04 12:38:31 Waiting for app: TestError to get deployed
2021/12/04 12:38:36 Waiting for app: TestError to get deployed
2021/12/04 12:38:41 Waiting for app: TestError to get deployed
2021/12/04 12:38:41 App: TestError got deployed
2021/12/04 12:38:41 Starting bucket ops to source bucket
2021/12/04 12:38:46 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/04 12:38:51 Waiting for dst bucket item count to get to: 300 curr count: 0
2021/12/04 12:38:51 src & dst bucket item count matched up. src bucket count: 300 dst bucket count: 300
2021/12/04 12:38:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:38:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:38:56 Waiting for app: TestError to get un-deployed
2021/12/04 12:39:01 Waiting for app: TestError to get un-deployed
2021/12/04 12:39:06 Waiting for app: TestError to get un-deployed
2021/12/04 12:39:11 Waiting for app: TestError to get un-deployed
2021/12/04 12:39:11 App: TestError got un-deployed
2021/12/04 12:39:11 No eventing-con process running
2021/12/04 12:39:11 Function: TestError delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestError deleting in the background"
}
2021/12/04 12:39:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:39:23 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:39:28 Waiting for dst bucket item count to get to: 0 curr count: 50
2021/12/04 12:39:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:39:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:39:40 Waiting for dst bucket item count to get to: 0 curr count: 185
2021/12/04 12:39:45 Waiting for dst bucket item count to get to: 0 curr count: 185
2021/12/04 12:39:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestError (108.86s)
=== RUN   TestCRLF
2021/12/04 12:39:55 Waiting for app: TestCRLF to get un-deployed
2021/12/04 12:39:55 App: TestCRLF got un-deployed
2021/12/04 12:39:55 No eventing-con process running
2021/12/04 12:39:55 Function: TestCRLF delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCRLF not found"
 }
}
2021/12/04 12:39:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:40:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:40:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:40:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:40:09 Deploying app: TestCRLF. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:40:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCRLF' in metakv",
  "warnings": null
 }
}
2021/12/04 12:40:15 Waiting for app: TestCRLF to get deployed
2021/12/04 12:40:20 Waiting for app: TestCRLF to get deployed
2021/12/04 12:40:25 Waiting for app: TestCRLF to get deployed
2021/12/04 12:40:30 Waiting for app: TestCRLF to get deployed
2021/12/04 12:40:30 App: TestCRLF got deployed
2021/12/04 12:40:30 Starting bucket ops to source bucket
2021/12/04 12:40:35 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:40:40 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 12:40:45 Waiting for dst bucket item count to get to: 100 curr count: 41
2021/12/04 12:40:45 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 12:40:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:40:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:40:50 Waiting for app: TestCRLF to get un-deployed
2021/12/04 12:40:55 Waiting for app: TestCRLF to get un-deployed
2021/12/04 12:41:00 Waiting for app: TestCRLF to get un-deployed
2021/12/04 12:41:05 Waiting for app: TestCRLF to get un-deployed
2021/12/04 12:41:05 App: TestCRLF got un-deployed
2021/12/04 12:41:05 No eventing-con process running
2021/12/04 12:41:05 Function: TestCRLF delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCRLF deleting in the background"
}
2021/12/04 12:41:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:41:17 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:41:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:41:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:41:29 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 12:41:34 Waiting for dst bucket item count to get to: 0 curr count: 41
2021/12/04 12:41:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCRLF (109.02s)
=== RUN   TestImportExport
2021/12/04 12:41:44 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:41:44 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 12:41:44 No eventing-con process running
2021/12/04 12:41:44 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_n1ql_insert_on_update not found"
 }
}
2021/12/04 12:41:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:41:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:41:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:41:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:42:03 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:42:03 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 12:42:03 No eventing-con process running
2021/12/04 12:42:03 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestImportExport_bucket_op_on_update not found"
 }
}
2021/12/04 12:42:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:42:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:42:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:42:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:42:17 Deploying app: TestImportExport_n1ql_insert_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:42:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_n1ql_insert_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 12:42:18 Deploying app: TestImportExport_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:42:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestImportExport_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 12:42:24 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 12:42:29 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 12:42:34 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 12:42:39 Waiting for app: TestImportExport_n1ql_insert_on_update to get deployed
2021/12/04 12:42:39 App: TestImportExport_n1ql_insert_on_update got deployed
2021/12/04 12:42:44 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 12:42:49 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 12:42:54 Waiting for app: TestImportExport_bucket_op_on_update to get deployed
2021/12/04 12:42:54 App: TestImportExport_bucket_op_on_update got deployed
2021/12/04 12:42:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:42:59 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:43:04 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:43:09 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:43:14 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:43:14 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 12:43:14 No eventing-con process running
2021/12/04 12:43:14 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/04 12:43:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:43:19 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:43:24 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:43:29 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:43:34 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:43:34 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 12:43:34 No eventing-con process running
2021/12/04 12:43:34 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/04 12:43:52 Waiting for app: TestImportExport_bucket_op_on_update to get un-deployed
2021/12/04 12:43:52 App: TestImportExport_bucket_op_on_update got un-deployed
2021/12/04 12:43:52 No eventing-con process running
2021/12/04 12:43:52 Function: TestImportExport_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_bucket_op_on_update deleting in the background"
}
2021/12/04 12:43:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:43:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:44:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:44:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:44:10 Waiting for app: TestImportExport_n1ql_insert_on_update to get un-deployed
2021/12/04 12:44:10 App: TestImportExport_n1ql_insert_on_update got un-deployed
2021/12/04 12:44:10 No eventing-con process running
2021/12/04 12:44:10 Function: TestImportExport_n1ql_insert_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestImportExport_n1ql_insert_on_update deleting in the background"
}
2021/12/04 12:44:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:44:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:44:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:44:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestImportExport (169.89s)
=== RUN   TestDeployUndeployLoopNonDefaultSettings
2021/12/04 12:44:34 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:44:34 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:44:34 No eventing-con process running
2021/12/04 12:44:34 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopNonDefaultSettings not found"
 }
}
2021/12/04 12:44:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:44:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:44:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:44:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:44:48 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:44:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:44:49 Starting bucket ops to source bucket
2021/12/04 12:44:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:45:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:45:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:45:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:45:17 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 12:45:17 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:45:22 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 12:45:22 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 12:45:22 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:45:22 Undeploying app: bucket_op_on_update
2021/12/04 12:45:22 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:45:27 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:45:32 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:45:37 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:45:42 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:45:42 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:45:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:45:42 No eventing-con process running
2021/12/04 12:45:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:45:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:46:26 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:46:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:46:27 Starting bucket ops to source bucket
2021/12/04 12:46:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:46:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:46:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:46:50 Waiting for dst bucket item count to get to: 5000 curr count: 661
2021/12/04 12:46:55 Waiting for dst bucket item count to get to: 5000 curr count: 661
2021/12/04 12:46:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:47:00 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 12:47:00 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 12:47:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:47:00 Undeploying app: bucket_op_on_update
2021/12/04 12:47:01 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:47:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:47:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:47:16 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:47:21 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:47:21 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:47:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:47:21 No eventing-con process running
2021/12/04 12:47:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:47:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:48:05 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:48:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:48:06 Starting bucket ops to source bucket
2021/12/04 12:48:14 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:48:19 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:48:24 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:48:29 Waiting for dst bucket item count to get to: 5000 curr count: 2253
2021/12/04 12:48:34 Waiting for dst bucket item count to get to: 5000 curr count: 2253
2021/12/04 12:48:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:48:39 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 12:48:39 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 12:48:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:48:39 Undeploying app: bucket_op_on_update
2021/12/04 12:48:39 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:48:44 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:48:49 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:48:54 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:48:59 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:48:59 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:48:59 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:48:59 No eventing-con process running
2021/12/04 12:49:01 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:49:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:49:43 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:49:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:49:44 Starting bucket ops to source bucket
2021/12/04 12:49:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:49:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:50:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:50:07 Waiting for dst bucket item count to get to: 5000 curr count: 640
2021/12/04 12:50:12 Waiting for dst bucket item count to get to: 5000 curr count: 3140
2021/12/04 12:50:12 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:50:17 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 12:50:17 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 12:50:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:50:17 Undeploying app: bucket_op_on_update
2021/12/04 12:50:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:50:23 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:50:28 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:50:33 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:50:38 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:50:38 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:50:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:50:38 No eventing-con process running
2021/12/04 12:50:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:51:22 Deploying app: TestDeployUndeployLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:51:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:51:23 Starting bucket ops to source bucket
2021/12/04 12:51:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:51:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:51:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:51:46 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 12:51:46 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:51:51 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get deployed
2021/12/04 12:51:51 App: TestDeployUndeployLoopNonDefaultSettings got deployed
2021/12/04 12:51:51 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:51:51 Undeploying app: bucket_op_on_update
2021/12/04 12:51:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:51:56 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:52:01 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:52:06 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:52:11 Waiting for app: TestDeployUndeployLoopNonDefaultSettings to get un-deployed
2021/12/04 12:52:11 App: TestDeployUndeployLoopNonDefaultSettings got un-deployed
2021/12/04 12:52:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:52:11 No eventing-con process running
2021/12/04 12:52:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:52:20 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:52:55 Function: TestDeployUndeployLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopNonDefaultSettings deleting in the background"
}
--- PASS: TestDeployUndeployLoopNonDefaultSettings (511.30s)
=== RUN   TestOnUpdateN1QLOp
2021/12/04 12:53:05 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 12:53:05 App: TestOnUpdateN1QLOp got un-deployed
2021/12/04 12:53:05 No eventing-con process running
2021/12/04 12:53:05 Function: TestOnUpdateN1QLOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateN1QLOp not found"
 }
}
2021/12/04 12:53:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:53:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:53:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:53:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:53:19 Deploying app: TestOnUpdateN1QLOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:53:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateN1QLOp' in metakv",
  "warnings": null
 }
}
2021/12/04 12:53:20 Starting bucket ops to source bucket
2021/12/04 12:53:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:53:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:53:38 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:53:43 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:53:48 Waiting for dst bucket item count to get to: 5000 curr count: 2513
2021/12/04 12:53:48 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:53:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:53:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:53:53 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 12:53:58 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 12:54:03 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 12:54:08 Waiting for app: TestOnUpdateN1QLOp to get un-deployed
2021/12/04 12:54:08 App: TestOnUpdateN1QLOp got un-deployed
2021/12/04 12:54:08 No eventing-con process running
2021/12/04 12:54:08 Function: TestOnUpdateN1QLOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateN1QLOp deleting in the background"
}
2021/12/04 12:54:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:54:20 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/04 12:54:25 Waiting for dst bucket item count to get to: 0 curr count: 3114
2021/12/04 12:54:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:54:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:54:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 12:54:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateN1QLOp (101.69s)
=== RUN   TestOnUpdateBucketOpDefaultSettings
2021/12/04 12:54:47 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 12:54:47 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/04 12:54:47 No eventing-con process running
2021/12/04 12:54:47 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings not found"
 }
}
2021/12/04 12:54:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:54:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:54:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:55:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:55:01 Deploying app: TestOnUpdateBucketOpDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:55:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:55:02 Starting bucket ops to source bucket
2021/12/04 12:55:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:55:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:55:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:55:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:55:25 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:55:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:55:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:55:30 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 12:55:35 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 12:55:40 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 12:55:45 Waiting for app: TestOnUpdateBucketOpDefaultSettings to get un-deployed
2021/12/04 12:55:45 App: TestOnUpdateBucketOpDefaultSettings got un-deployed
2021/12/04 12:55:45 No eventing-con process running
2021/12/04 12:55:45 Function: TestOnUpdateBucketOpDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings deleting in the background"
}
2021/12/04 12:55:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:55:57 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 12:55:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:55:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:56:09 Waiting for dst bucket item count to get to: 0 curr count: 4590
2021/12/04 12:56:14 Waiting for dst bucket item count to get to: 0 curr count: 2090
2021/12/04 12:56:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings (96.77s)
=== RUN   TestOnUpdateBucketOpNonDefaultSettings
2021/12/04 12:56:24 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 12:56:24 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 12:56:24 No eventing-con process running
2021/12/04 12:56:24 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/04 12:56:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:56:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:56:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:56:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:56:38 Deploying app: TestOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:56:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 12:56:39 Starting bucket ops to source bucket
2021/12/04 12:56:47 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:56:52 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:56:57 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:57:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:57:07 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 12:57:07 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:57:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:57:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:57:12 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 12:57:17 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 12:57:22 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 12:57:27 Waiting for app: TestOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 12:57:27 App: TestOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 12:57:27 No eventing-con process running
2021/12/04 12:57:27 Function: TestOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/04 12:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:57:39 Waiting for dst bucket item count to get to: 0 curr count: 4764
2021/12/04 12:57:44 Waiting for dst bucket item count to get to: 0 curr count: 2264
2021/12/04 12:57:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:57:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:57:56 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 12:57:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpNonDefaultSettings (101.98s)
=== RUN   TestOnUpdateBucketOpDefaultSettings10K
2021/12/04 12:58:06 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 12:58:06 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/04 12:58:06 No eventing-con process running
2021/12/04 12:58:06 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings10K not found"
 }
}
2021/12/04 12:58:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:58:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:58:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:58:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:58:19 Deploying app: TestOnUpdateBucketOpDefaultSettings10K. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:58:21 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings10K' in metakv",
  "warnings": null
 }
}
2021/12/04 12:58:21 Starting bucket ops to source bucket
2021/12/04 12:58:29 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:58:34 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:58:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:58:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 12:58:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 12:58:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 12:58:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 12:58:49 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 12:58:54 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 12:58:59 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 12:59:04 Waiting for app: TestOnUpdateBucketOpDefaultSettings10K to get un-deployed
2021/12/04 12:59:04 App: TestOnUpdateBucketOpDefaultSettings10K got un-deployed
2021/12/04 12:59:04 No eventing-con process running
2021/12/04 12:59:04 Function: TestOnUpdateBucketOpDefaultSettings10K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings10K deleting in the background"
}
2021/12/04 12:59:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:59:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 12:59:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:59:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:59:28 Waiting for dst bucket item count to get to: 0 curr count: 2764
2021/12/04 12:59:33 Waiting for dst bucket item count to get to: 0 curr count: 264
2021/12/04 12:59:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings10K (97.03s)
=== RUN   TestOnUpdateBucketOpDefaultSettings100K
2021/12/04 12:59:43 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 12:59:43 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/04 12:59:43 No eventing-con process running
2021/12/04 12:59:43 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateBucketOpDefaultSettings100K not found"
 }
}
2021/12/04 12:59:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:59:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:59:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 12:59:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 12:59:56 Deploying app: TestOnUpdateBucketOpDefaultSettings100K. Item count src bucket: 0 dst bucket: 0
2021/12/04 12:59:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateBucketOpDefaultSettings100K' in metakv",
  "warnings": null
 }
}
2021/12/04 12:59:58 Starting bucket ops to source bucket
2021/12/04 13:00:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:00:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:00:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:00:21 Waiting for dst bucket item count to get to: 5000 curr count: 299
2021/12/04 13:00:26 Waiting for dst bucket item count to get to: 5000 curr count: 299
2021/12/04 13:00:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:00:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:00:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:00:31 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 13:00:36 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 13:00:41 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 13:00:46 Waiting for app: TestOnUpdateBucketOpDefaultSettings100K to get un-deployed
2021/12/04 13:00:46 App: TestOnUpdateBucketOpDefaultSettings100K got un-deployed
2021/12/04 13:00:46 No eventing-con process running
2021/12/04 13:00:46 Function: TestOnUpdateBucketOpDefaultSettings100K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateBucketOpDefaultSettings100K deleting in the background"
}
2021/12/04 13:00:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:00:58 Waiting for dst bucket item count to get to: 0 curr count: 3132
2021/12/04 13:01:03 Waiting for dst bucket item count to get to: 0 curr count: 632
2021/12/04 13:01:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:01:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:01:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:01:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateBucketOpDefaultSettings100K (102.27s)
=== RUN   TestOnDeleteBucketOp
2021/12/04 13:01:25 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 13:01:25 App: TestOnDeleteBucketOp got un-deployed
2021/12/04 13:01:25 No eventing-con process running
2021/12/04 13:01:25 Function: TestOnDeleteBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp not found"
 }
}
2021/12/04 13:01:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:01:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:01:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:01:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:01:39 Deploying app: TestOnDeleteBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:01:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 13:01:40 Starting bucket ops to source bucket
2021/12/04 13:01:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:01:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:02:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:02:06 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/04 13:02:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:02:06 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:02:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:02:11 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 13:02:16 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 13:02:21 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 13:02:26 Waiting for app: TestOnDeleteBucketOp to get un-deployed
2021/12/04 13:02:26 App: TestOnDeleteBucketOp got un-deployed
2021/12/04 13:02:26 No eventing-con process running
2021/12/04 13:02:26 Function: TestOnDeleteBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp deleting in the background"
}
2021/12/04 13:02:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:02:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:02:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:02:45 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:02:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp (90.05s)
=== RUN   TestOnDeleteBucketOp5K
2021/12/04 13:02:55 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 13:02:55 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/04 13:02:55 No eventing-con process running
2021/12/04 13:02:55 Function: TestOnDeleteBucketOp5K delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteBucketOp5K not found"
 }
}
2021/12/04 13:02:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:03:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:03:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:03:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:03:09 Deploying app: TestOnDeleteBucketOp5K. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:03:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteBucketOp5K' in metakv",
  "warnings": null
 }
}
2021/12/04 13:03:10 Starting bucket ops to source bucket
2021/12/04 13:03:21 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:03:26 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:03:31 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:03:36 Waiting for dst bucket item count to get to: 5000 curr count: 2504
2021/12/04 13:03:36 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:03:36 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:03:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:03:41 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 13:03:46 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 13:03:51 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 13:03:56 Waiting for app: TestOnDeleteBucketOp5K to get un-deployed
2021/12/04 13:03:56 App: TestOnDeleteBucketOp5K got un-deployed
2021/12/04 13:03:56 No eventing-con process running
2021/12/04 13:03:56 Function: TestOnDeleteBucketOp5K delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteBucketOp5K deleting in the background"
}
2021/12/04 13:03:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:04:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:04:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:04:15 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:04:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteBucketOp5K (90.20s)
=== RUN   TestTimerBucketOp
2021/12/04 13:04:25 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 13:04:25 App: TestTimerBucketOp got un-deployed
2021/12/04 13:04:25 No eventing-con process running
2021/12/04 13:04:25 Function: TestTimerBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerBucketOp not found"
 }
}
2021/12/04 13:04:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:04:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:04:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:04:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:04:39 Deploying app: TestTimerBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:04:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 13:04:40 Starting bucket ops to source bucket
2021/12/04 13:04:48 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:04:53 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:04:58 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:05:03 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:05:08 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:05:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:05:18 Waiting for dst bucket item count to get to: 5000 curr count: 828
2021/12/04 13:05:23 Waiting for dst bucket item count to get to: 5000 curr count: 3328
2021/12/04 13:05:23 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:05:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:05:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:05:28 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 13:05:33 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 13:05:38 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 13:05:43 Waiting for app: TestTimerBucketOp to get un-deployed
2021/12/04 13:05:43 App: TestTimerBucketOp got un-deployed
2021/12/04 13:05:43 No eventing-con process running
2021/12/04 13:05:43 Function: TestTimerBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerBucketOp deleting in the background"
}
2021/12/04 13:05:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:05:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:05:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:05:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:06:07 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:06:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerBucketOp (111.51s)
=== RUN   TestTimerInPastBucketOp
2021/12/04 13:06:17 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 13:06:17 App: TestTimerInPastBucketOp got un-deployed
2021/12/04 13:06:17 No eventing-con process running
2021/12/04 13:06:17 Function: TestTimerInPastBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerInPastBucketOp not found"
 }
}
2021/12/04 13:06:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:06:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:06:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:06:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:06:30 Deploying app: TestTimerInPastBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:06:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerInPastBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 13:06:32 Starting bucket ops to source bucket
2021/12/04 13:06:39 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:06:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:06:49 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:06:54 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:06:59 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:07:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:07:10 Waiting for dst bucket item count to get to: 5000 curr count: 1401
2021/12/04 13:07:15 Waiting for dst bucket item count to get to: 5000 curr count: 1401
2021/12/04 13:07:15 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:07:15 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:07:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:07:20 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 13:07:25 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 13:07:30 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 13:07:35 Waiting for app: TestTimerInPastBucketOp to get un-deployed
2021/12/04 13:07:35 App: TestTimerInPastBucketOp got un-deployed
2021/12/04 13:07:35 No eventing-con process running
2021/12/04 13:07:35 Function: TestTimerInPastBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerInPastBucketOp deleting in the background"
}
2021/12/04 13:07:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:07:46 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:07:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:07:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:07:58 Waiting for dst bucket item count to get to: 0 curr count: 4265
2021/12/04 13:08:03 Waiting for dst bucket item count to get to: 0 curr count: 1765
2021/12/04 13:08:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerInPastBucketOp (116.59s)
=== RUN   TestDeployUndeployLoopTimer
2021/12/04 13:08:13 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:08:13 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:08:13 No eventing-con process running
2021/12/04 13:08:13 Function: TestDeployUndeployLoopTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeployUndeployLoopTimer not found"
 }
}
2021/12/04 13:08:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:08:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:08:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:08:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:08:27 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:08:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 13:08:28 Starting bucket ops to source bucket
2021/12/04 13:08:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:08:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:08:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:08:51 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:08:56 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:09:01 Waiting for dst bucket item count to get to: 5000 curr count: 147
2021/12/04 13:09:06 Waiting for dst bucket item count to get to: 5000 curr count: 2647
2021/12/04 13:09:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:09:11 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 13:09:11 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 13:09:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:09:11 Undeploying app: bucket_op_with_timer
2021/12/04 13:09:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:09:16 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:09:21 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:09:26 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:09:31 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:09:31 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:09:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:09:31 No eventing-con process running
2021/12/04 13:09:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:09:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:10:16 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:10:17 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 13:10:17 Starting bucket ops to source bucket
2021/12/04 13:10:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:10:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:11:00 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 13:11:00 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 13:11:00 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:11:00 Undeploying app: bucket_op_with_timer
2021/12/04 13:11:00 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:11:05 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:11:10 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:11:15 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:11:20 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:11:20 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:11:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:11:20 No eventing-con process running
2021/12/04 13:11:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:11:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:12:04 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:12:05 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 13:12:05 Starting bucket ops to source bucket
2021/12/04 13:12:13 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:12:18 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:12:23 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:12:28 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:12:33 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:12:38 Waiting for dst bucket item count to get to: 5000 curr count: 53
2021/12/04 13:12:43 Waiting for dst bucket item count to get to: 5000 curr count: 53
2021/12/04 13:12:43 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:12:48 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 13:12:48 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 13:12:48 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:12:48 Undeploying app: bucket_op_with_timer
2021/12/04 13:12:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:12:53 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:12:58 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:13:03 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:13:08 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:13:08 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:13:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:13:08 No eventing-con process running
2021/12/04 13:13:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:13:52 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:13:54 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 13:13:54 Starting bucket ops to source bucket
2021/12/04 13:14:02 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:07 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:12 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:17 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:22 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:27 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:32 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:14:37 Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 13:14:37 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:14:42 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 13:14:42 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 13:14:42 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:14:42 Undeploying app: bucket_op_with_timer
2021/12/04 13:14:42 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:14:47 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:14:52 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:14:57 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:15:02 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:15:02 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:15:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:15:02 No eventing-con process running
2021/12/04 13:15:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:15:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:15:46 Deploying app: TestDeployUndeployLoopTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:15:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeployUndeployLoopTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 13:15:47 Starting bucket ops to source bucket
2021/12/04 13:15:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:16:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:16:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:16:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:16:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:16:20 Waiting for dst bucket item count to get to: 5000 curr count: 486
2021/12/04 13:16:25 Waiting for dst bucket item count to get to: 5000 curr count: 486
2021/12/04 13:16:30 Waiting for dst bucket item count to get to: 5000 curr count: 2858
2021/12/04 13:16:35 Waiting for dst bucket item count to get to: 5000 curr count: 2858
2021/12/04 13:16:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:16:40 Waiting for app: TestDeployUndeployLoopTimer to get deployed
2021/12/04 13:16:40 App: TestDeployUndeployLoopTimer got deployed
2021/12/04 13:16:40 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:16:40 Undeploying app: bucket_op_with_timer
2021/12/04 13:16:40 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:16:45 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:16:50 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:16:55 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:17:00 Waiting for app: TestDeployUndeployLoopTimer to get un-deployed
2021/12/04 13:17:00 App: TestDeployUndeployLoopTimer got un-deployed
2021/12/04 13:17:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:17:00 No eventing-con process running
2021/12/04 13:17:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:17:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:17:45 Function: TestDeployUndeployLoopTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeployUndeployLoopTimer deleting in the background"
}
--- PASS: TestDeployUndeployLoopTimer (581.17s)
=== RUN   TestMultipleHandlers
2021/12/04 13:17:55 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 13:17:55 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/04 13:17:55 No eventing-con process running
2021/12/04 13:17:55 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_update not found"
 }
}
2021/12/04 13:17:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:18:01 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:18:03 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:18:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:18:13 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 13:18:13 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/04 13:18:13 No eventing-con process running
2021/12/04 13:18:13 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMultipleHandlers_bucket_op_on_delete not found"
 }
}
2021/12/04 13:18:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:18:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:18:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:18:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:18:27 Deploying app: TestMultipleHandlers_bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:18:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 13:18:28 Deploying app: TestMultipleHandlers_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:18:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMultipleHandlers_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/04 13:18:34 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 13:18:39 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 13:18:44 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 13:18:49 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get deployed
2021/12/04 13:18:49 App: TestMultipleHandlers_bucket_op_on_update got deployed
2021/12/04 13:18:54 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 13:18:59 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 13:19:04 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get deployed
2021/12/04 13:19:04 App: TestMultipleHandlers_bucket_op_on_delete got deployed
2021/12/04 13:19:04 Starting bucket ops to source bucket
2021/12/04 13:19:14 Waiting for dst bucket item count to get to: 5000 curr count: 1422
2021/12/04 13:19:19 Waiting for dst bucket item count to get to: 5000 curr count: 3906
2021/12/04 13:19:24 Waiting for dst bucket item count to get to: 5000 curr count: 4984
2021/12/04 13:19:24 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:19:24 Starting bucket ops to source bucket
2021/12/04 13:19:40 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:19:45 Waiting for dst bucket item count to get to: 10000 curr count: 7504
2021/12/04 13:19:45 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 13:19:45 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:19:45 Function: TestMultipleHandlers_bucket_op_on_update update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:19:45 Function: TestMultipleHandlers_bucket_op_on_delete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:19:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:19:50 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 13:19:55 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 13:20:00 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 13:20:05 Waiting for app: TestMultipleHandlers_bucket_op_on_update to get un-deployed
2021/12/04 13:20:05 App: TestMultipleHandlers_bucket_op_on_update got un-deployed
2021/12/04 13:20:05 No eventing-con process running
2021/12/04 13:20:05 Function: TestMultipleHandlers_bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_update deleting in the background"
}
2021/12/04 13:20:07 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:20:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:20:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:20:24 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 13:20:29 Waiting for dst bucket item count to get to: 0 curr count: 4996
2021/12/04 13:20:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:20:29 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:20:34 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 13:20:39 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 13:20:44 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 13:20:49 Waiting for app: TestMultipleHandlers_bucket_op_on_delete to get un-deployed
2021/12/04 13:20:49 App: TestMultipleHandlers_bucket_op_on_delete got un-deployed
2021/12/04 13:20:49 No eventing-con process running
2021/12/04 13:20:49 Function: TestMultipleHandlers_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMultipleHandlers_bucket_op_on_delete deleting in the background"
}
2021/12/04 13:20:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:20:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:20:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:21:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMultipleHandlers (198.46s)
=== RUN   TestPauseResumeLoopDefaultSettings
2021/12/04 13:21:13 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 13:21:13 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/04 13:21:13 No eventing-con process running
2021/12/04 13:21:13 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopDefaultSettings not found"
 }
}
2021/12/04 13:21:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:21:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:21:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:21:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:21:27 Deploying app: TestPauseResumeLoopDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:21:28 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 13:21:28 Starting bucket ops to source bucket
2021/12/04 13:21:36 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:21:41 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:21:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:21:51 Waiting for dst bucket item count to get to: 5000 curr count: 22
2021/12/04 13:21:56 Waiting for dst bucket item count to get to: 5000 curr count: 22
2021/12/04 13:21:56 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:21:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:22:01 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 13:22:01 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 13:22:01 Pausing app: bucket_op_on_update
2021/12/04 13:22:01 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:22:06 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 13:22:06 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 13:22:06 Resuming app: bucket_op_on_update
2021/12/04 13:22:07 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:22:07 Starting bucket ops to source bucket
2021/12/04 13:22:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:22:20 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:22:25 Waiting for dst bucket item count to get to: 10000 curr count: 7500
2021/12/04 13:22:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 13:22:25 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:22:30 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 13:22:30 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 13:22:30 Pausing app: bucket_op_on_update
2021/12/04 13:22:30 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:22:35 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 13:22:35 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 13:22:35 Resuming app: bucket_op_on_update
2021/12/04 13:22:36 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:22:36 Starting bucket ops to source bucket
2021/12/04 13:22:44 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:22:49 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:22:54 Waiting for dst bucket item count to get to: 15000 curr count: 12500
2021/12/04 13:22:54 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/04 13:22:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:23:00 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 13:23:00 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 13:23:00 Pausing app: bucket_op_on_update
2021/12/04 13:23:00 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:23:05 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 13:23:05 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 13:23:05 Resuming app: bucket_op_on_update
2021/12/04 13:23:06 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:23:06 Starting bucket ops to source bucket
2021/12/04 13:23:14 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:23:19 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:23:24 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/04 13:23:29 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/04 13:23:29 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 13:23:29 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:23:34 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 13:23:34 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 13:23:34 Pausing app: bucket_op_on_update
2021/12/04 13:23:34 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:23:39 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 13:23:39 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 13:23:39 Resuming app: bucket_op_on_update
2021/12/04 13:23:40 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:23:40 Starting bucket ops to source bucket
2021/12/04 13:23:48 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:23:53 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:23:58 Waiting for dst bucket item count to get to: 25000 curr count: 20926
2021/12/04 13:24:03 Waiting for dst bucket item count to get to: 25000 curr count: 23426
2021/12/04 13:24:03 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 13:24:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:24:08 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to deployed
2021/12/04 13:24:08 Function: TestPauseResumeLoopDefaultSettings status changed to deployed
2021/12/04 13:24:08 Pausing app: bucket_op_on_update
2021/12/04 13:24:08 Function: TestPauseResumeLoopDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:24:13 Waiting for function: TestPauseResumeLoopDefaultSettings status to change to paused
2021/12/04 13:24:13 Function: TestPauseResumeLoopDefaultSettings status changed to paused
2021/12/04 13:24:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:24:18 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 13:24:23 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 13:24:28 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 13:24:33 Waiting for app: TestPauseResumeLoopDefaultSettings to get un-deployed
2021/12/04 13:24:33 App: TestPauseResumeLoopDefaultSettings got un-deployed
2021/12/04 13:24:33 No eventing-con process running
2021/12/04 13:24:33 Function: TestPauseResumeLoopDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopDefaultSettings deleting in the background"
}
2021/12/04 13:24:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:24:45 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 13:24:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:24:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:24:57 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 13:24:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopDefaultSettings (234.12s)
=== RUN   TestPauseResumeLoopNonDefaultSettings
2021/12/04 13:25:07 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 13:25:07 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/04 13:25:07 No eventing-con process running
2021/12/04 13:25:07 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeLoopNonDefaultSettings not found"
 }
}
2021/12/04 13:25:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:25:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:25:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:25:21 Deploying app: TestPauseResumeLoopNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:25:22 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeLoopNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 13:25:22 Starting bucket ops to source bucket
2021/12/04 13:25:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:25:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:25:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:25:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:25:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:25:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:26:00 Waiting for dst bucket item count to get to: 5000 curr count: 2073
2021/12/04 13:26:05 Waiting for dst bucket item count to get to: 5000 curr count: 2073
2021/12/04 13:26:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:26:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:26:10 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 13:26:10 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 13:26:10 Pausing app: bucket_op_with_timer
2021/12/04 13:26:10 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:26:15 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 13:26:15 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 13:26:15 Resuming app: bucket_op_with_timer
2021/12/04 13:26:16 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:26:16 Starting bucket ops to source bucket
2021/12/04 13:26:24 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:26:29 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:26:34 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:26:39 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:26:44 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:26:49 Waiting for dst bucket item count to get to: 10000 curr count: 8049
2021/12/04 13:26:54 Waiting for dst bucket item count to get to: 10000 curr count: 8049
2021/12/04 13:26:54 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 13:26:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:26:59 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 13:26:59 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 13:26:59 Pausing app: bucket_op_with_timer
2021/12/04 13:27:00 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:27:05 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 13:27:05 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 13:27:05 Resuming app: bucket_op_with_timer
2021/12/04 13:27:06 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:27:06 Starting bucket ops to source bucket
2021/12/04 13:27:14 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:27:19 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:27:24 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:27:29 Waiting for dst bucket item count to get to: 15000 curr count: 10000
2021/12/04 13:27:34 Waiting for dst bucket item count to get to: 15000 curr count: 10818
2021/12/04 13:27:39 Waiting for dst bucket item count to get to: 15000 curr count: 13318
2021/12/04 13:27:39 src & dst bucket item count matched up. src bucket count: 15000 dst bucket count: 15000
2021/12/04 13:27:39 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:27:44 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 13:27:44 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 13:27:44 Pausing app: bucket_op_with_timer
2021/12/04 13:27:44 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:27:49 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 13:27:49 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 13:27:49 Resuming app: bucket_op_with_timer
2021/12/04 13:27:50 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:27:50 Starting bucket ops to source bucket
2021/12/04 13:27:58 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:03 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:08 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:13 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:18 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:23 Waiting for dst bucket item count to get to: 20000 curr count: 15000
2021/12/04 13:28:28 Waiting for dst bucket item count to get to: 20000 curr count: 17500
2021/12/04 13:28:28 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 13:28:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:28:33 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 13:28:33 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 13:28:33 Pausing app: bucket_op_with_timer
2021/12/04 13:28:33 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:28:38 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 13:28:38 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 13:28:38 Resuming app: bucket_op_with_timer
2021/12/04 13:28:39 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:28:39 Starting bucket ops to source bucket
2021/12/04 13:28:47 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:28:52 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:28:57 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:29:02 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:29:07 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:29:12 Waiting for dst bucket item count to get to: 25000 curr count: 20000
2021/12/04 13:29:17 Waiting for dst bucket item count to get to: 25000 curr count: 22500
2021/12/04 13:29:17 src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 13:29:17 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:29:22 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to deployed
2021/12/04 13:29:22 Function: TestPauseResumeLoopNonDefaultSettings status changed to deployed
2021/12/04 13:29:22 Pausing app: bucket_op_with_timer
2021/12/04 13:29:22 Function: TestPauseResumeLoopNonDefaultSettings update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:4 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:77 timer_context_size:15728640 user_prefix:eventing worker_count:4] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:29:27 Waiting for function: TestPauseResumeLoopNonDefaultSettings status to change to paused
2021/12/04 13:29:27 Function: TestPauseResumeLoopNonDefaultSettings status changed to paused
2021/12/04 13:29:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:29:33 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 13:29:38 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 13:29:43 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 13:29:48 Waiting for app: TestPauseResumeLoopNonDefaultSettings to get un-deployed
2021/12/04 13:29:48 App: TestPauseResumeLoopNonDefaultSettings got un-deployed
2021/12/04 13:29:48 No eventing-con process running
2021/12/04 13:29:48 Function: TestPauseResumeLoopNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeLoopNonDefaultSettings deleting in the background"
}
2021/12/04 13:29:49 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:29:59 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/04 13:30:04 Waiting for dst bucket item count to get to: 0 curr count: 12500
2021/12/04 13:30:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:30:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:30:16 Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 13:30:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeLoopNonDefaultSettings (319.12s)
=== RUN   TestPauseAndResumeWithWorkerCountChange
2021/12/04 13:30:26 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 13:30:26 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/04 13:30:26 No eventing-con process running
2021/12/04 13:30:26 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseAndResumeWithWorkerCountChange not found"
 }
}
2021/12/04 13:30:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:30:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:30:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:30:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:30:40 Deploying app: TestPauseAndResumeWithWorkerCountChange. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:30:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseAndResumeWithWorkerCountChange' in metakv",
  "warnings": null
 }
}
2021/12/04 13:30:46 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 13:30:51 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 13:30:56 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 13:31:01 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get deployed
2021/12/04 13:31:01 App: TestPauseAndResumeWithWorkerCountChange got deployed
2021/12/04 13:31:01 Starting bucket ops to source bucket
2021/12/04 13:31:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:31:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:31:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:31:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:31:30 Waiting for dst bucket item count to get to: 5000 curr count: 2817
2021/12/04 13:31:35 Waiting for dst bucket item count to get to: 5000 curr count: 2817
2021/12/04 13:31:35 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:31:35 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:31:35 Pausing app: bucket_op_with_timer
2021/12/04 13:31:36 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:31:41 Waiting for function: TestPauseAndResumeWithWorkerCountChange status to change to paused
2021/12/04 13:31:41 Function: TestPauseAndResumeWithWorkerCountChange status changed to paused
2021/12/04 13:31:41 Resuming app: bucket_op_with_timer
2021/12/04 13:31:42 Function: TestPauseAndResumeWithWorkerCountChange update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:31:42 Starting bucket ops to source bucket
2021/12/04 13:31:53 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:31:58 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:32:03 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:32:08 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:32:14 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:32:19 Waiting for dst bucket item count to get to: 10000 curr count: 9982
2021/12/04 13:32:24 Waiting for dst bucket item count to get to: 10000 curr count: 9982
2021/12/04 13:32:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 13:32:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:32:29 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 13:32:34 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 13:32:39 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 13:32:44 Waiting for app: TestPauseAndResumeWithWorkerCountChange to get un-deployed
2021/12/04 13:32:44 App: TestPauseAndResumeWithWorkerCountChange got un-deployed
2021/12/04 13:32:44 No eventing-con process running
2021/12/04 13:32:44 Function: TestPauseAndResumeWithWorkerCountChange delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseAndResumeWithWorkerCountChange deleting in the background"
}
2021/12/04 13:32:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:32:55 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 13:32:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:32:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:33:07 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 13:33:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseAndResumeWithWorkerCountChange (171.08s)
=== RUN   TestPauseResumeWithEventingReb
2021/12/04 13:33:17 Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 13:33:17 App: TestPauseResumeWithEventingReb got un-deployed
2021/12/04 13:33:17 No eventing-con process running
2021/12/04 13:33:17 Function: TestPauseResumeWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestPauseResumeWithEventingReb not found"
 }
}
2021/12/04 13:33:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:33:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:33:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:33:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:33:31 Deploying app: TestPauseResumeWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:33:32 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestPauseResumeWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 13:33:37 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 13:33:42 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 13:33:47 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 13:33:52 Waiting for app: TestPauseResumeWithEventingReb to get deployed
2021/12/04 13:33:52 App: TestPauseResumeWithEventingReb got deployed
2021/12/04 13:33:52 Starting bucket ops to source bucket
2021/12/04 13:34:01 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:34:06 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:34:11 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:34:16 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:34:21 Waiting for dst bucket item count to get to: 5000 curr count: 4195
2021/12/04 13:34:26 Waiting for dst bucket item count to get to: 5000 curr count: 4195
2021/12/04 13:34:26 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:34:26 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:34:26 Pausing app: bucket_op_with_timer
2021/12/04 13:34:27 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:34:32 Waiting for function: TestPauseResumeWithEventingReb status to change to paused
2021/12/04 13:34:32 Function: TestPauseResumeWithEventingReb status changed to paused
2021/12/04 13:34:32 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 13:34:32 Removing node(s): [] from the cluster
2021/12/04 13:34:37 Rebalance progress: 29.16666666666666
2021/12/04 13:34:42 Rebalance progress: 29.16666666666666
2021/12/04 13:34:47 Rebalance progress: 100
2021/12/04 13:34:47 Resuming app: bucket_op_with_timer
2021/12/04 13:34:48 Function: TestPauseResumeWithEventingReb update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:true sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:6] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:34:48 Starting bucket ops to source bucket
2021/12/04 13:35:00 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:35:05 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:35:10 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:35:15 Waiting for dst bucket item count to get to: 10000 curr count: 5000
2021/12/04 13:35:20 Waiting for dst bucket item count to get to: 10000 curr count: 5023
2021/12/04 13:35:25 Waiting for dst bucket item count to get to: 10000 curr count: 7523
2021/12/04 13:35:25 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 13:35:25 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 13:35:30 Rebalance progress: 29.16666666666666
2021/12/04 13:35:35 Rebalance progress: 29.16666666666666
2021/12/04 13:35:40 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/04 13:35:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:36:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:36:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:36:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:36:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:37:00 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:37:15 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:37:30 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 13:37:45 waitForIndexes waiting for indexes: [{Name:#primary Bucket:default Completion:100} {Name:#primary Bucket:eventing Completion:72} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: a47b3cd7b6fcba347c941e3c44f95133
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [171-227]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [228-284]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 57
	v: [285-341]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [0-56]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [57-113]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [114-170]
Producer node: 172.31.2.134:9300	Node UUID: c6c72bf434ed0a3c7eb5d258bdbe00f0
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [968-1023]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [683-739]
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [740-796]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [797-853]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [854-910]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [911-967]
Producer node: 127.0.0.1:9303	Node UUID: 29beee3f904fdf467b8a8a471bd9ecf8
	workerID: worker_TestPauseResumeWithEventingReb_1
	len: 57
	v: [399-455]
	workerID: worker_TestPauseResumeWithEventingReb_2
	len: 57
	v: [456-512]
	workerID: worker_TestPauseResumeWithEventingReb_3
	len: 57
	v: [513-569]
	workerID: worker_TestPauseResumeWithEventingReb_4
	len: 57
	v: [570-626]
	workerID: worker_TestPauseResumeWithEventingReb_5
	len: 56
	v: [627-682]
	workerID: worker_TestPauseResumeWithEventingReb_0
	len: 57
	v: [342-398]


2021/12/04 13:38:01 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:38:01 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:38:06 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 13:38:11 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 13:38:16 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 13:38:21 clusterSetup.go:397: Waiting for app: TestPauseResumeWithEventingReb to get un-deployed
2021/12/04 13:38:21 clusterSetup.go:405: App: TestPauseResumeWithEventingReb got un-deployed
2021/12/04 13:38:21 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:38:21 eventingSetup.go:418: Function: TestPauseResumeWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestPauseResumeWithEventingReb deleting in the background"
}
2021/12/04 13:38:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:38:33 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 13:38:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:38:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:38:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:38:50 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:38:55 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:38:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestPauseResumeWithEventingReb (347.24s)
=== RUN   TestBucketDeleteAfterPause
2021/12/04 13:39:05 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/04 13:39:05 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/04 13:39:05 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:39:05 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteAfterPause not found"
 }
}
2021/12/04 13:39:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:39:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:39:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:39:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:39:18 eventingSetup.go:125: Deploying app: TestBucketDeleteAfterPause. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:39:19 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteAfterPause' in metakv",
  "warnings": null
 }
}
2021/12/04 13:39:24 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 13:39:29 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 13:39:34 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 13:39:39 clusterSetup.go:346: Waiting for app: TestBucketDeleteAfterPause to get deployed
2021/12/04 13:39:39 clusterSetup.go:354: App: TestBucketDeleteAfterPause got deployed
2021/12/04 13:39:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:39:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:39:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 13:39:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 13:39:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:39:59 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:39:59 handler_test.go:712: Pausing app: bucket_op_on_update
2021/12/04 13:39:59 eventingSetup.go:270: Function: TestBucketDeleteAfterPause update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:40:04 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to paused
2021/12/04 13:40:04 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to paused
2021/12/04 13:40:04 handler_test.go:716: Deleting source bucket: default
2021/12/04 13:40:11 eventingSetup.go:719: Waiting for function: TestBucketDeleteAfterPause status to change to undeployed
2021/12/04 13:40:11 eventingSetup.go:727: Function: TestBucketDeleteAfterPause status changed to undeployed
2021/12/04 13:40:16 clusterSetup.go:397: Waiting for app: TestBucketDeleteAfterPause to get un-deployed
2021/12/04 13:40:16 clusterSetup.go:405: App: TestBucketDeleteAfterPause got un-deployed
2021/12/04 13:40:16 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:40:16 eventingSetup.go:418: Function: TestBucketDeleteAfterPause delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteAfterPause deleting in the background"
}
2021/12/04 13:40:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:40:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:40:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:40:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:40:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:40:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:40:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketDeleteAfterPause (109.95s)
=== RUN   TestChangeFnCodeBetweenPauseResume
2021/12/04 13:40:54 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 13:40:54 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/04 13:40:55 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:40:55 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestChangeFnCodeBetweenPauseResume not found"
 }
}
2021/12/04 13:40:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:41:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:41:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:41:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:41:08 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:41:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/04 13:41:14 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 13:41:19 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 13:41:24 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 13:41:29 clusterSetup.go:346: Waiting for app: TestChangeFnCodeBetweenPauseResume to get deployed
2021/12/04 13:41:29 clusterSetup.go:354: App: TestChangeFnCodeBetweenPauseResume got deployed
2021/12/04 13:41:29 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:42:04 handler_test.go:739: Pausing function: TestChangeFnCodeBetweenPauseResume
2021/12/04 13:42:04 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:true description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:42:09 eventingSetup.go:719: Waiting for function: TestChangeFnCodeBetweenPauseResume status to change to paused
2021/12/04 13:42:09 eventingSetup.go:727: Function: TestChangeFnCodeBetweenPauseResume status changed to paused
2021/12/04 13:45:09 handler_test.go:748: Resuming function: TestChangeFnCodeBetweenPauseResume
2021/12/04 13:45:09 eventingSetup.go:125: Deploying app: TestChangeFnCodeBetweenPauseResume. Item count src bucket: 5000 dst bucket: 0
2021/12/04 13:45:10 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestChangeFnCodeBetweenPauseResume' in metakv",
  "warnings": null
 }
}
2021/12/04 13:45:15 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 13:45:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:45:15 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 0 expected 5000
2021/12/04 13:45:20 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 13:45:20 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:45:20 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 3176 expected 5000
2021/12/04 13:45:25 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 13:45:25 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:45:25 eventingSetup.go:799: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 4872 expected 5000
2021/12/04 13:45:30 eventingSetup.go:795: Waiting for function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter to get to 5000
2021/12/04 13:45:30 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:45:30 eventingSetup.go:804: Function: TestChangeFnCodeBetweenPauseResume stat: timer_callback_missing_counter got to 5000
2021/12/04 13:45:30 handler_test.go:753: Undeploying function: TestChangeFnCodeBetweenPauseResume
2021/12/04 13:45:30 eventingSetup.go:270: Function: TestChangeFnCodeBetweenPauseResume update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:45:40 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 13:45:45 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 13:45:50 clusterSetup.go:397: Waiting for app: TestChangeFnCodeBetweenPauseResume to get un-deployed
2021/12/04 13:45:50 clusterSetup.go:405: App: TestChangeFnCodeBetweenPauseResume got un-deployed
2021/12/04 13:45:50 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:45:50 eventingSetup.go:418: Function: TestChangeFnCodeBetweenPauseResume delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestChangeFnCodeBetweenPauseResume deleting in the background"
}
2021/12/04 13:45:52 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:46:02 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:46:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:46:07 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:46:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:46:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestChangeFnCodeBetweenPauseResume (329.04s)
=== RUN   TestCommentUnCommentOnDelete
2021/12/04 13:46:24 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 13:46:24 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/04 13:46:24 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:46:24 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCommentUnCommentOnDelete not found"
 }
}
2021/12/04 13:46:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:46:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:46:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:46:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:46:37 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:46:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 13:46:43 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:46:48 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:46:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:46:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:46:58 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/04 13:46:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:47:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:47:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 799
2021/12/04 13:47:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 799
2021/12/04 13:47:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:47:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:47:17 handler_test.go:780: Undeploying app: TestCommentUnCommentOnDelete
2021/12/04 13:47:17 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:47:47 eventingSetup.go:125: Deploying app: TestCommentUnCommentOnDelete. Item count src bucket: 5000 dst bucket: 5000
2021/12/04 13:47:48 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCommentUnCommentOnDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 13:47:53 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:47:58 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:48:03 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:48:08 clusterSetup.go:346: Waiting for app: TestCommentUnCommentOnDelete to get deployed
2021/12/04 13:48:08 clusterSetup.go:354: App: TestCommentUnCommentOnDelete got deployed
2021/12/04 13:48:08 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:48:21 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:48:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:48:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:48:26 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:48:26 handler_test.go:799: Undeploying app: TestCommentUnCommentOnDelete
2021/12/04 13:48:26 eventingSetup.go:270: Function: TestCommentUnCommentOnDelete update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:48:36 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 13:48:41 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 13:48:46 clusterSetup.go:397: Waiting for app: TestCommentUnCommentOnDelete to get un-deployed
2021/12/04 13:48:46 clusterSetup.go:405: App: TestCommentUnCommentOnDelete got un-deployed
2021/12/04 13:48:46 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:48:46 eventingSetup.go:418: Function: TestCommentUnCommentOnDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCommentUnCommentOnDelete deleting in the background"
}
2021/12/04 13:48:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:48:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:48:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:48:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCommentUnCommentOnDelete (165.92s)
=== RUN   TestCPPWorkerCleanup
2021/12/04 13:49:09 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 13:49:09 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/04 13:49:09 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:49:09 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCPPWorkerCleanup not found"
 }
}
2021/12/04 13:49:11 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:49:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:49:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:49:23 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:49:23 eventingSetup.go:125: Deploying app: TestCPPWorkerCleanup. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:49:24 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCPPWorkerCleanup' in metakv",
  "warnings": null
 }
}
2021/12/04 13:49:24 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:49:31 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:49:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:49:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:49:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 165
2021/12/04 13:49:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 1923
2021/12/04 13:49:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 4258
2021/12/04 13:49:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:49:56 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:49:56 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:50:01 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 13:50:06 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 13:50:11 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 13:50:16 clusterSetup.go:397: Waiting for app: TestCPPWorkerCleanup to get un-deployed
2021/12/04 13:50:16 clusterSetup.go:405: App: TestCPPWorkerCleanup got un-deployed
2021/12/04 13:50:16 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:50:16 eventingSetup.go:418: Function: TestCPPWorkerCleanup delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCPPWorkerCleanup deleting in the background"
}
2021/12/04 13:50:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:50:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:50:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:50:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:50:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:50:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:50:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCPPWorkerCleanup (134.74s)
=== RUN   TestWithUserXattrs
2021/12/04 13:51:24 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 13:51:24 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/04 13:51:24 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:51:24 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestWithUserXattrs not found"
 }
}
2021/12/04 13:51:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:51:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:51:32 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:51:37 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:51:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:51:39 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:51:41 eventingSetup.go:125: Deploying app: TestWithUserXattrs. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:51:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestWithUserXattrs' in metakv",
  "warnings": null
 }
}
2021/12/04 13:51:48 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 13:51:53 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 13:51:58 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 13:52:03 clusterSetup.go:346: Waiting for app: TestWithUserXattrs to get deployed
2021/12/04 13:52:03 clusterSetup.go:354: App: TestWithUserXattrs got deployed
2021/12/04 13:52:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:52:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:52:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 13:52:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:52:17 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:52:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 13:52:22 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 13:52:27 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 13:52:32 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 13:52:37 clusterSetup.go:397: Waiting for app: TestWithUserXattrs to get un-deployed
2021/12/04 13:52:37 clusterSetup.go:405: App: TestWithUserXattrs got un-deployed
2021/12/04 13:52:37 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:52:37 eventingSetup.go:418: Function: TestWithUserXattrs delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestWithUserXattrs deleting in the background"
}
2021/12/04 13:52:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:52:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:52:54 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:52:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:52:55 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:53:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:53:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestWithUserXattrs (111.14s)
=== RUN   TestEventProcessingPostBucketFlush
2021/12/04 13:53:15 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:53:15 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 13:53:15 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:53:15 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventProcessingPostBucketFlush not found"
 }
}
2021/12/04 13:53:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:53:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:53:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:53:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:53:29 eventingSetup.go:125: Deploying app: TestEventProcessingPostBucketFlush. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:53:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventProcessingPostBucketFlush' in metakv",
  "warnings": null
 }
}
2021/12/04 13:53:35 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 13:53:40 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 13:53:45 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 13:53:50 clusterSetup.go:346: Waiting for app: TestEventProcessingPostBucketFlush to get deployed
2021/12/04 13:53:50 clusterSetup.go:354: App: TestEventProcessingPostBucketFlush got deployed
2021/12/04 13:53:50 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:53:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:54:04 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 13:54:09 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 2500
2021/12/04 13:54:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 13:54:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:54:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:54:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4632
2021/12/04 13:54:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4632
2021/12/04 13:54:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:54:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:54:39 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 13:54:44 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 13:54:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:54:44 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 13:55:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17272
2021/12/04 13:55:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 25000 curr count: 17272
2021/12/04 13:55:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 25000 dst bucket count: 25000
2021/12/04 13:55:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 13:55:15 eventingSetup.go:270: Function: TestEventProcessingPostBucketFlush update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 13:55:20 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:55:25 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:55:30 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:55:35 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:55:35 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 13:55:35 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:55:40 clusterSetup.go:397: Waiting for app: TestEventProcessingPostBucketFlush to get un-deployed
2021/12/04 13:55:40 clusterSetup.go:405: App: TestEventProcessingPostBucketFlush got un-deployed
2021/12/04 13:55:40 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:55:40 eventingSetup.go:418: Function: TestEventProcessingPostBucketFlush delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventProcessingPostBucketFlush deleting in the background"
}
2021/12/04 13:55:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:55:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 13:55:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 25000
2021/12/04 13:55:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:55:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:56:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 17651
2021/12/04 13:56:13 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5151
2021/12/04 13:56:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventProcessingPostBucketFlush (187.69s)
=== RUN   TestMetaBucketDelete
2021/12/04 13:56:23 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 13:56:23 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 13:56:23 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:56:23 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDelete not found"
 }
}
2021/12/04 13:56:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:56:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:56:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:56:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:56:36 eventingSetup.go:125: Deploying app: TestMetaBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:56:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 13:56:42 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 13:56:47 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 13:56:52 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 13:56:57 clusterSetup.go:346: Waiting for app: TestMetaBucketDelete to get deployed
2021/12/04 13:56:57 clusterSetup.go:354: App: TestMetaBucketDelete got deployed
2021/12/04 13:56:57 handler_test.go:902: Deleting metadata bucket: eventing
2021/12/04 13:56:58 handler_test.go:904: Deleted metadata bucket: eventing
2021/12/04 13:57:03 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 13:57:03 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 13:57:18 clusterSetup.go:397: Waiting for app: TestMetaBucketDelete to get un-deployed
2021/12/04 13:57:18 clusterSetup.go:405: App: TestMetaBucketDelete got un-deployed
2021/12/04 13:57:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:57:18 eventingSetup.go:418: Function: TestMetaBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDelete deleting in the background"
}
2021/12/04 13:57:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:57:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:57:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:57:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDelete (78.43s)
=== RUN   TestMetaBucketDeleteWithBootstrap
2021/12/04 13:57:41 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 13:57:41 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 13:57:41 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:57:41 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 13:57:43 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:57:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:57:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:57:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:57:55 eventingSetup.go:125: Deploying app: TestMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:57:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 13:58:15 handler_test.go:923: Deleting metadata bucket: eventing
2021/12/04 13:58:15 handler_test.go:925: Deleted metadata bucket: eventing
2021/12/04 13:58:30 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 13:58:30 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 13:58:45 clusterSetup.go:397: Waiting for app: TestMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 13:58:45 clusterSetup.go:405: App: TestMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 13:58:45 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:58:45 eventingSetup.go:418: Function: TestMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 13:58:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:58:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:58:53 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:58:58 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaBucketDeleteWithBootstrap (86.94s)
=== RUN   TestSourceBucketDelete
2021/12/04 13:59:08 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 13:59:08 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 13:59:08 eventingSetup.go:46: No eventing-con process running
2021/12/04 13:59:08 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDelete not found"
 }
}
2021/12/04 13:59:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:59:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:59:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 13:59:22 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 13:59:22 eventingSetup.go:125: Deploying app: TestSourceBucketDelete. Item count src bucket: 0 dst bucket: 0
2021/12/04 13:59:23 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDelete' in metakv",
  "warnings": null
 }
}
2021/12/04 13:59:28 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 13:59:33 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 13:59:38 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 13:59:43 clusterSetup.go:346: Waiting for app: TestSourceBucketDelete to get deployed
2021/12/04 13:59:43 clusterSetup.go:354: App: TestSourceBucketDelete got deployed
2021/12/04 13:59:43 handler_test.go:945: Deleting source bucket: default
2021/12/04 13:59:44 handler_test.go:947: Deleted source bucket: default
2021/12/04 13:59:49 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 13:59:49 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 14:00:04 clusterSetup.go:397: Waiting for app: TestSourceBucketDelete to get un-deployed
2021/12/04 14:00:04 clusterSetup.go:405: App: TestSourceBucketDelete got un-deployed
2021/12/04 14:00:04 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:00:04 eventingSetup.go:418: Function: TestSourceBucketDelete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDelete deleting in the background"
}
2021/12/04 14:00:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:00:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:00:12 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:00:17 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDelete (79.01s)
=== RUN   TestSourceBucketDeleteWithBootstrap
2021/12/04 14:00:27 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:00:27 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:00:27 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:00:27 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 14:00:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:00:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:00:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:00:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:00:41 eventingSetup.go:125: Deploying app: TestSourceBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:00:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 14:01:01 handler_test.go:966: Deleting source bucket: default
2021/12/04 14:01:02 handler_test.go:968: Deleted source bucket: default
2021/12/04 14:01:17 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:01:17 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:01:32 clusterSetup.go:397: Waiting for app: TestSourceBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:01:32 clusterSetup.go:405: App: TestSourceBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:01:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:01:32 eventingSetup.go:418: Function: TestSourceBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 14:01:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:01:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:01:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:01:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceBucketDeleteWithBootstrap (87.78s)
=== RUN   TestSourceAndMetaBucketDeleteWithBootstrap
2021/12/04 14:01:55 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:01:55 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:01:55 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:01:55 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap not found"
 }
}
2021/12/04 14:01:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:02:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:02:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:02:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:02:08 eventingSetup.go:125: Deploying app: TestSourceAndMetaBucketDeleteWithBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:02:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestSourceAndMetaBucketDeleteWithBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 14:02:28 handler_test.go:988: Deleting source bucket: default
2021/12/04 14:02:30 handler_test.go:990: Deleted source bucket: default
2021/12/04 14:02:30 handler_test.go:992: Deleting metadata bucket: eventing
2021/12/04 14:02:30 handler_test.go:994: Deleted metadata bucket: eventing
2021/12/04 14:02:45 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:02:45 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:03:00 clusterSetup.go:397: Waiting for app: TestSourceAndMetaBucketDeleteWithBootstrap to get un-deployed
2021/12/04 14:03:00 clusterSetup.go:405: App: TestSourceAndMetaBucketDeleteWithBootstrap got un-deployed
2021/12/04 14:03:00 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:03:00 eventingSetup.go:418: Function: TestSourceAndMetaBucketDeleteWithBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestSourceAndMetaBucketDeleteWithBootstrap deleting in the background"
}
2021/12/04 14:03:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:03:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:03:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:03:13 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestSourceAndMetaBucketDeleteWithBootstrap (88.10s)
=== RUN   TestUndeployDuringBootstrap
2021/12/04 14:03:23 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 14:03:23 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 14:03:23 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:03:23 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployDuringBootstrap not found"
 }
}
2021/12/04 14:03:25 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:03:30 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:03:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:03:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:03:36 eventingSetup.go:125: Deploying app: TestUndeployDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:03:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 14:03:38 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 14:03:50 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:03:50 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:03:50 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/04 14:03:50 clusterSetup.go:378: App: TestUndeployDuringBootstrap started bootstrap
2021/12/04 14:03:50 clusterSetup.go:375: Apps undergoing bootstrap: true
2021/12/04 14:03:55 clusterSetup.go:367: No apps undergoing bootstrap
2021/12/04 14:04:00 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 14:04:05 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 14:04:10 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 14:04:10 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 14:04:10 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:04:35 clusterSetup.go:397: Waiting for app: TestUndeployDuringBootstrap to get un-deployed
2021/12/04 14:04:35 clusterSetup.go:405: App: TestUndeployDuringBootstrap got un-deployed
2021/12/04 14:04:35 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:04:35 eventingSetup.go:418: Function: TestUndeployDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployDuringBootstrap deleting in the background"
}
2021/12/04 14:04:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:04:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:04:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:04:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:04:58 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3028
2021/12/04 14:05:03 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 528
2021/12/04 14:05:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployDuringBootstrap (109.52s)
=== RUN   TestDeleteBeforeUndeploy
2021/12/04 14:05:08 eventingSetup.go:125: Deploying app: TestDeleteBeforeUndeploy. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:05:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDeleteBeforeUndeploy' in metakv",
  "warnings": null
 }
}
2021/12/04 14:05:14 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 14:05:19 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 14:05:24 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 14:05:29 clusterSetup.go:346: Waiting for app: TestDeleteBeforeUndeploy to get deployed
2021/12/04 14:05:29 clusterSetup.go:354: App: TestDeleteBeforeUndeploy got deployed
2021/12/04 14:05:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 422 dump: {
 "name": "ERR_APP_NOT_UNDEPLOYED",
 "code": 22,
 "description": "Function hasn't undeployed",
 "attributes": null,
 "runtime_info": {
  "code": 22,
  "info": "Function: TestDeleteBeforeUndeploy skipping delete request from primary store, as it hasn't been undeployed"
 }
}
2021/12/04 14:05:29 eventingSetup.go:270: Function: TestDeleteBeforeUndeploy update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 14:05:29 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDeleteBeforeUndeploy deleting in the background"
}
2021/12/04 14:05:34 clusterSetup.go:397: Waiting for app: TestDeleteBeforeUndeploy to get un-deployed
2021/12/04 14:05:34 clusterSetup.go:405: App: TestDeleteBeforeUndeploy got un-deployed
2021/12/04 14:05:34 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:05:34 eventingSetup.go:418: Function: TestDeleteBeforeUndeploy delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDeleteBeforeUndeploy not found"
 }
}
2021/12/04 14:05:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:05:41 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:05:42 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:05:47 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDeleteBeforeUndeploy (44.61s)
=== RUN   TestUndeployWhenTimersAreFired
2021/12/04 14:05:57 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:05:57 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 14:05:57 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:05:57 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWhenTimersAreFired not found"
 }
}
2021/12/04 14:05:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:06:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:06:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:06:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:06:11 eventingSetup.go:125: Deploying app: TestUndeployWhenTimersAreFired. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:06:12 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWhenTimersAreFired' in metakv",
  "warnings": null
 }
}
2021/12/04 14:06:17 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 14:06:22 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 14:06:27 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 14:06:32 clusterSetup.go:346: Waiting for app: TestUndeployWhenTimersAreFired to get deployed
2021/12/04 14:06:32 clusterSetup.go:354: App: TestUndeployWhenTimersAreFired got deployed
2021/12/04 14:06:32 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 14:07:02 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:07:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:07:12 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:07:17 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:07:22 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:07:22 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 14:07:22 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:09:07 clusterSetup.go:397: Waiting for app: TestUndeployWhenTimersAreFired to get un-deployed
2021/12/04 14:09:07 clusterSetup.go:405: App: TestUndeployWhenTimersAreFired got un-deployed
2021/12/04 14:09:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:09:07 eventingSetup.go:418: Function: TestUndeployWhenTimersAreFired delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWhenTimersAreFired deleting in the background"
}
2021/12/04 14:09:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:09:19 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:09:24 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 14:09:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:09:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:09:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:09:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWhenTimersAreFired (228.16s)
=== RUN   TestUndeployWithKVFailover
2021/12/04 14:09:46 clusterSetup.go:397: Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 14:09:46 clusterSetup.go:405: App: TestUndeployWithKVFailover got un-deployed
2021/12/04 14:09:46 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:09:46 eventingSetup.go:418: Function: TestUndeployWithKVFailover delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployWithKVFailover not found"
 }
}
2021/12/04 14:09:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:09:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:09:54 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:09:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:09:59 eventingSetup.go:125: Deploying app: TestUndeployWithKVFailover. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:10:00 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployWithKVFailover' in metakv",
  "warnings": null
 }
}
2021/12/04 14:10:00 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 14:10:07 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:12 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:17 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:22 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:32 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:10:37 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3405
2021/12/04 14:10:42 eventingSetup.go:441: Waiting for dst bucket item count to get to: 5000 curr count: 3405
2021/12/04 14:10:42 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 14:10:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 14:10:43 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 14:10:48 Rebalance progress: 29.16666666666666
2021/12/04 14:10:53 Rebalance progress: 29.16666666666666
2021/12/04 14:10:58 Rebalance progress: 100
2021/12/04 14:10:58 Failing over: [127.0.0.1:19003]
2021/12/04 14:11:08 Function: TestUndeployWithKVFailover update settings: map[bucket_cache_age:1000 bucket_cache_size:67108864 cpp_worker_thread_count:1 dcp_stream_boundary:everything deployment_status:false description:Sample app execution_timeout:5 language_compatibility:6.5.0 lcb_inst_capacity:5 log_level:INFO n1ql_consistency:request num_timer_partitions:128 processing_status:false sock_batch_size:1 timer_context_size:15728640 user_prefix:eventing worker_count:3] requested, response code: 0 dump: &{Name: Code:0 Description: Attributes:[] RuntimeInfo: httpResponseCode:0}
2021/12/04 14:12:08 Removing node(s): [] from the cluster
2021/12/04 14:12:13 Rebalance progress: 29.16666666666666
2021/12/04 14:12:18 Rebalance progress: 29.16666666666666
2021/12/04 14:12:23 Rebalance progress: 100
2021/12/04 14:12:28 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 14:12:28 App: TestUndeployWithKVFailover got un-deployed
2021/12/04 14:12:28 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:12:33 Waiting for app: TestUndeployWithKVFailover to get un-deployed
2021/12/04 14:12:33 App: TestUndeployWithKVFailover got un-deployed
2021/12/04 14:12:33 No eventing-con process running
2021/12/04 14:12:33 Function: TestUndeployWithKVFailover delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployWithKVFailover deleting in the background"
}
2021/12/04 14:12:34 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:12:44 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:12:49 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 14:12:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:12:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:13:01 Waiting for dst bucket item count to get to: 0 curr count: 4697
2021/12/04 14:13:06 Waiting for dst bucket item count to get to: 0 curr count: 4697
2021/12/04 14:13:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestUndeployWithKVFailover (210.37s)
=== RUN   TestBucketFlushWhileFnDeployed
2021/12/04 14:13:16 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 14:13:16 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/04 14:13:16 No eventing-con process running
2021/12/04 14:13:16 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketFlushWhileFnDeployed not found"
 }
}
2021/12/04 14:13:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:13:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:13:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:13:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:13:29 Deploying app: TestBucketFlushWhileFnDeployed. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:13:30 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketFlushWhileFnDeployed' in metakv",
  "warnings": null
 }
}
2021/12/04 14:13:35 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 14:13:40 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 14:13:45 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 14:13:50 Waiting for app: TestBucketFlushWhileFnDeployed to get deployed
2021/12/04 14:13:50 App: TestBucketFlushWhileFnDeployed got deployed
2021/12/04 14:13:50 Starting bucket ops to source bucket
2021/12/04 14:14:11 Waiting for dst bucket item count to get to: 20000 curr count: 9830
2021/12/04 14:14:11 src & dst bucket item count matched up. src bucket count: 20000 dst bucket count: 20000
2021/12/04 14:14:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:14:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:14:24 Waiting for dst bucket item count to get to: 0 curr count: 20000
2021/12/04 14:14:29 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 14:14:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:14:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:14:40 Waiting for dst bucket item count to get to: 0 curr count: 15615
2021/12/04 14:14:45 Waiting for dst bucket item count to get to: 0 curr count: 15615
2021/12/04 14:14:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:14:45 Starting bucket ops to source bucket
2021/12/04 14:14:58 Waiting for dst bucket item count to get to: 10000 curr count: 3393
2021/12/04 14:15:03 Waiting for dst bucket item count to get to: 10000 curr count: 3393
2021/12/04 14:15:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 14:15:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:15:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:15:09 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 14:15:14 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 14:15:19 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 14:15:24 Waiting for app: TestBucketFlushWhileFnDeployed to get un-deployed
2021/12/04 14:15:24 App: TestBucketFlushWhileFnDeployed got un-deployed
2021/12/04 14:15:24 No eventing-con process running
2021/12/04 14:15:24 Function: TestBucketFlushWhileFnDeployed delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketFlushWhileFnDeployed deleting in the background"
}
2021/12/04 14:15:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:15:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 14:15:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:15:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:15:47 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 14:15:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBucketFlushWhileFnDeployed (160.71s)
=== RUN   TestTimerOverwrite
2021/12/04 14:15:57 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 14:15:57 App: TestTimerOverwrite got un-deployed
2021/12/04 14:15:57 No eventing-con process running
2021/12/04 14:15:57 Function: TestTimerOverwrite delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverwrite not found"
 }
}
2021/12/04 14:15:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:16:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:16:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:16:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:16:10 Deploying app: TestTimerOverwrite. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:16:11 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverwrite' in metakv",
  "warnings": null
 }
}
2021/12/04 14:16:16 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 14:16:21 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 14:16:26 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 14:16:31 Waiting for app: TestTimerOverwrite to get deployed
2021/12/04 14:16:31 App: TestTimerOverwrite got deployed
2021/12/04 14:16:41 Starting bucket ops to source bucket
2021/12/04 14:16:50 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:16:55 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:00 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:05 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:10 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:15 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:20 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:25 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:30 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:35 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:40 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:45 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:17:50 Waiting for dst bucket item count to get to: 5000 curr count: 3487
2021/12/04 14:17:55 Waiting for dst bucket item count to get to: 5000 curr count: 3487
2021/12/04 14:17:55 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 14:18:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:18:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:18:10 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 14:18:15 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 14:18:20 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 14:18:25 Waiting for app: TestTimerOverwrite to get un-deployed
2021/12/04 14:18:25 App: TestTimerOverwrite got un-deployed
2021/12/04 14:18:25 No eventing-con process running
2021/12/04 14:18:25 Function: TestTimerOverwrite delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverwrite deleting in the background"
}
2021/12/04 14:18:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:18:36 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:18:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:18:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:18:48 Waiting for dst bucket item count to get to: 0 curr count: 4139
2021/12/04 14:18:53 Waiting for dst bucket item count to get to: 0 curr count: 1654
2021/12/04 14:18:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverwrite (186.69s)
=== RUN   TestUndeployBackdoorDuringBootstrap
2021/12/04 14:18:58 Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 14:18:59 Removing node(s): [] from the cluster
2021/12/04 14:19:04 Rebalance progress: 100
2021/12/04 14:19:04 Failing over: [127.0.0.1:19003]
2021/12/04 14:19:09 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 14:19:09 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 14:19:09 No eventing-con process running
2021/12/04 14:19:09 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/04 14:19:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:19:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:19:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:19:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:19:22 Deploying app: TestUndeployBackdoorDuringBootstrap. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:19:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestUndeployBackdoorDuringBootstrap' in metakv",
  "warnings": null
 }
}
2021/12/04 14:19:23 Starting bucket ops to source bucket
2021/12/04 14:19:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:19:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:19:33 Context: Post to set retry counter, response code: 200 dump: 
2021/12/04 14:19:33 set retry counter, response &{[] }
2021/12/04 14:20:38 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 14:20:38 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 14:20:38 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:20:38 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestUndeployBackdoorDuringBootstrap deleting in the background"
}
2021/12/04 14:20:43 Waiting for app: TestUndeployBackdoorDuringBootstrap to get un-deployed
2021/12/04 14:20:43 App: TestUndeployBackdoorDuringBootstrap got un-deployed
2021/12/04 14:20:43 No eventing-con process running
2021/12/04 14:20:43 Function: TestUndeployBackdoorDuringBootstrap delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestUndeployBackdoorDuringBootstrap not found"
 }
}
2021/12/04 14:20:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:20:55 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:20:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:20:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:21:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:21:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:21:06 Removing node(s): [] from the cluster
2021/12/04 14:21:12 Rebalance progress: 100
--- PASS: TestUndeployBackdoorDuringBootstrap (138.24s)
=== RUN   TestOnUpdateSrcMutation
2021/12/04 14:21:22 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 14:21:22 App: TestOnUpdateSrcMutation got un-deployed
2021/12/04 14:21:22 No eventing-con process running
2021/12/04 14:21:22 Function: TestOnUpdateSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutation not found"
 }
}
2021/12/04 14:21:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:21:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:21:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:21:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:21:35 Deploying app: TestOnUpdateSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:21:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/04 14:21:36 Starting bucket ops to source bucket
2021/12/04 14:22:03 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 14:22:03 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:22:03 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:22:08 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 14:22:13 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 14:22:18 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 14:22:23 Waiting for app: TestOnUpdateSrcMutation to get un-deployed
2021/12/04 14:22:23 App: TestOnUpdateSrcMutation got un-deployed
2021/12/04 14:22:23 No eventing-con process running
2021/12/04 14:22:23 Function: TestOnUpdateSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutation deleting in the background"
}
2021/12/04 14:22:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:22:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 14:22:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:22:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:22:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutation (89.79s)
=== RUN   TestOnDeleteSrcMutation
2021/12/04 14:22:51 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 14:22:51 App: TestOnDeleteSrcMutation got un-deployed
2021/12/04 14:22:51 No eventing-con process running
2021/12/04 14:22:51 Function: TestOnDeleteSrcMutation delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutation not found"
 }
}
2021/12/04 14:22:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:22:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:22:59 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:23:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:23:05 Deploying app: TestOnDeleteSrcMutation. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:23:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutation' in metakv",
  "warnings": null
 }
}
2021/12/04 14:23:06 Starting bucket ops to source bucket
2021/12/04 14:23:34 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 14:23:34 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:23:34 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:23:39 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 14:23:44 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 14:23:49 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 14:23:54 Waiting for app: TestOnDeleteSrcMutation to get un-deployed
2021/12/04 14:23:55 App: TestOnDeleteSrcMutation got un-deployed
2021/12/04 14:23:55 No eventing-con process running
2021/12/04 14:23:55 Function: TestOnDeleteSrcMutation delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutation deleting in the background"
}
2021/12/04 14:23:56 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:24:06 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:24:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:24:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:24:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutation (91.34s)
=== RUN   TestOnUpdateSrcMutationWithTimer
2021/12/04 14:24:23 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 14:24:23 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/04 14:24:23 No eventing-con process running
2021/12/04 14:24:23 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnUpdateSrcMutationWithTimer not found"
 }
}
2021/12/04 14:24:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:24:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:24:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:24:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:24:36 Deploying app: TestOnUpdateSrcMutationWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:24:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnUpdateSrcMutationWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 14:24:37 Starting bucket ops to source bucket
2021/12/04 14:25:24 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 14:25:24 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:25:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:25:29 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 14:25:34 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 14:25:39 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 14:25:44 Waiting for app: TestOnUpdateSrcMutationWithTimer to get un-deployed
2021/12/04 14:25:44 App: TestOnUpdateSrcMutationWithTimer got un-deployed
2021/12/04 14:25:44 No eventing-con process running
2021/12/04 14:25:44 Function: TestOnUpdateSrcMutationWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnUpdateSrcMutationWithTimer deleting in the background"
}
2021/12/04 14:25:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:25:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 14:25:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:25:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:26:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnUpdateSrcMutationWithTimer (109.68s)
=== RUN   TestOnDeleteSrcMutationsWithTimer
2021/12/04 14:26:12 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 14:26:12 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/04 14:26:12 No eventing-con process running
2021/12/04 14:26:12 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteSrcMutationsWithTimer not found"
 }
}
2021/12/04 14:26:14 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:26:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:26:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:26:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:26:26 Deploying app: TestOnDeleteSrcMutationsWithTimer. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:26:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteSrcMutationsWithTimer' in metakv",
  "warnings": null
 }
}
2021/12/04 14:26:27 Starting bucket ops to source bucket
2021/12/04 14:27:05 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 14:27:05 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:27:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:27:11 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 14:27:16 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 14:27:21 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 14:27:26 Waiting for app: TestOnDeleteSrcMutationsWithTimer to get un-deployed
2021/12/04 14:27:26 App: TestOnDeleteSrcMutationsWithTimer got un-deployed
2021/12/04 14:27:26 No eventing-con process running
2021/12/04 14:27:26 Function: TestOnDeleteSrcMutationsWithTimer delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteSrcMutationsWithTimer deleting in the background"
}
2021/12/04 14:27:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:27:37 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 14:27:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:27:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:27:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteSrcMutationsWithTimer (101.45s)
=== RUN   TestInterHandlerRecursion
2021/12/04 14:27:54 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:27:54 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 14:27:54 No eventing-con process running
2021/12/04 14:27:54 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/04 14:27:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:28:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:28:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:28:12 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:28:12 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 14:28:12 No eventing-con process running
2021/12/04 14:28:12 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 14:28:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:28:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:28:20 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:28:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:28:25 Deploying app: TestInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:28:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/04 14:28:26 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:28:31 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:28:36 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:28:41 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:28:47 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:28:47 App: TestInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/04 14:28:47 Deploying app: TestInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:28:47 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
}
2021/12/04 14:28:47 response body {
 "name": "ERR_INTER_FUNCTION_RECURSION",
 "code": 50,
 "description": "Inter function recursion error, only one function is allowed to do source bucket mutation/delete on a bucket",
 "attributes": null,
 "runtime_info": {
  "code": 50,
  "info": "Inter handler recursion error"
 }
} err 
2021/12/04 14:28:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:28:52 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:28:57 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:29:02 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:29:07 Waiting for app: TestInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:29:07 App: TestInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 14:29:07 No eventing-con process running
2021/12/04 14:29:07 Function: TestInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/04 14:29:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:29:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:29:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:29:20 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:29:25 Waiting for app: TestInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:29:25 App: TestInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 14:29:25 No eventing-con process running
2021/12/04 14:29:25 Function: TestInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 14:29:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:29:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:29:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:29:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterHandlerRecursion (114.18s)
=== RUN   TestInterBucketRecursion
2021/12/04 14:29:43 Deploying app: TestInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:29:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 14:29:44 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:29:44 Deploying app: TestInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:29:44 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion_function2 causes a cycle involving functions: [TestInterBucketRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/04 14:29:49 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 14:29:54 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 14:29:59 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 14:30:04 Waiting for app: TestInterBucketRecursion_function1 to get deployed
2021/12/04 14:30:04 App: TestInterBucketRecursion_function1 got deployed
2021/12/04 14:30:04 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:30:09 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:30:14 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:30:19 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:30:24 Waiting for app: TestInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:30:24 App: TestInterBucketRecursion_function1 got un-deployed
2021/12/04 14:30:24 No eventing-con process running
2021/12/04 14:30:24 Function: TestInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion_function1 deleting in the background"
}
2021/12/04 14:30:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:30:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:30:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:30:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:30:42 Waiting for app: TestInterBucketRecursion_function2 to get un-deployed
2021/12/04 14:30:42 App: TestInterBucketRecursion_function2 got un-deployed
2021/12/04 14:30:42 No eventing-con process running
2021/12/04 14:30:42 Function: TestInterBucketRecursion_function2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion_function2 not found"
 }
}
2021/12/04 14:30:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:30:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:30:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:30:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion (77.64s)
=== RUN   TestLargeHandler
2021/12/04 14:30:56 Changed force_compress value to false
2021/12/04 14:30:56 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:30:56 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_APPCODE_SIZE",
 "code": 39,
 "description": "Function Code size is more than the configured limit",
 "attributes": null,
 "runtime_info": {
  "code": 39,
  "info": "Function: TestLargeHandler_function handler Code size is more than 131072. Code Size: 131768"
 }
}
2021/12/04 14:30:56 Changed force_compress value to true
2021/12/04 14:30:56 Deploying app: TestLargeHandler_function. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:30:58 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestLargeHandler_function' in metakv",
  "warnings": null
 }
}
2021/12/04 14:31:03 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 14:31:08 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 14:31:13 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 14:31:18 Waiting for app: TestLargeHandler_function to get deployed
2021/12/04 14:31:18 App: TestLargeHandler_function got deployed
2021/12/04 14:31:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:31:23 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 14:31:28 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 14:31:33 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 14:31:38 Waiting for app: TestLargeHandler_function to get un-deployed
2021/12/04 14:31:38 App: TestLargeHandler_function got un-deployed
2021/12/04 14:31:38 No eventing-con process running
2021/12/04 14:31:38 Function: TestLargeHandler_function delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestLargeHandler_function deleting in the background"
}
2021/12/04 14:31:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:31:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:31:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:31:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestLargeHandler (55.44s)
=== RUN   TestAllowInterHandlerRecursion
2021/12/04 14:32:01 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:32:01 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 14:32:01 No eventing-con process running
2021/12/04 14:32:01 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op not found"
 }
}
2021/12/04 14:32:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:32:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:32:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:32:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:32:19 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:32:19 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 14:32:19 No eventing-con process running
2021/12/04 14:32:19 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete not found"
 }
}
2021/12/04 14:32:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:32:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:32:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:32:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:32:32 Deploying app: TestAllowInterHandlerRecursion_source_bucket_update_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:32:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
}
2021/12/04 14:32:34 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_source_bucket_update_op' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:32:39 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:32:44 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:32:49 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:32:54 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get deployed
2021/12/04 14:32:54 App: TestAllowInterHandlerRecursion_source_bucket_update_op got deployed
2021/12/04 14:32:54 Deploying app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:32:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
}
2021/12/04 14:32:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterHandlerRecursion_src_bucket_op_on_delete' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:33:00 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 14:33:05 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 14:33:10 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 14:33:15 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get deployed
2021/12/04 14:33:15 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got deployed
2021/12/04 14:33:15 TestAllowInterHandlerRecursion_src_bucket_op_on_delete is deployed
2021/12/04 14:33:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:33:20 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:33:25 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:33:30 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:33:35 Waiting for app: TestAllowInterHandlerRecursion_source_bucket_update_op to get un-deployed
2021/12/04 14:33:35 App: TestAllowInterHandlerRecursion_source_bucket_update_op got un-deployed
2021/12/04 14:33:35 No eventing-con process running
2021/12/04 14:33:35 Function: TestAllowInterHandlerRecursion_source_bucket_update_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_source_bucket_update_op deleting in the background"
}
2021/12/04 14:33:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:33:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:33:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:33:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:33:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:33:54 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:33:59 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:34:04 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:34:09 Waiting for app: TestAllowInterHandlerRecursion_src_bucket_op_on_delete to get un-deployed
2021/12/04 14:34:09 App: TestAllowInterHandlerRecursion_src_bucket_op_on_delete got un-deployed
2021/12/04 14:34:09 No eventing-con process running
2021/12/04 14:34:09 Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterHandlerRecursion_src_bucket_op_on_delete deleting in the background"
}
2021/12/04 14:34:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:34:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:34:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:34:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterHandlerRecursion (151.66s)
=== RUN   TestAllowInterBucketRecursion
2021/12/04 14:34:28 Deploying app: TestAllowInterBucketRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:34:29 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 14:34:29 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:34:34 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 14:34:39 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 14:34:44 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 14:34:49 Waiting for app: TestAllowInterBucketRecursion_function1 to get deployed
2021/12/04 14:34:49 App: TestAllowInterBucketRecursion_function1 got deployed
2021/12/04 14:34:49 Deploying app: TestAllowInterBucketRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:34:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 14:34:50 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestAllowInterBucketRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:34:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 14:35:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 14:35:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 14:35:10 Waiting for app: TestAllowInterBucketRecursion_function2 to get deployed
2021/12/04 14:35:10 App: TestAllowInterBucketRecursion_function2 got deployed
2021/12/04 14:35:10 TestAllowInterBucketRecursion_function2 is deployed
2021/12/04 14:35:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:35:15 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:35:20 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:35:25 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:35:30 Waiting for app: TestAllowInterBucketRecursion_function1 to get un-deployed
2021/12/04 14:35:30 App: TestAllowInterBucketRecursion_function1 got un-deployed
2021/12/04 14:35:30 No eventing-con process running
2021/12/04 14:35:30 Function: TestAllowInterBucketRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function1 deleting in the background"
}
2021/12/04 14:35:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:35:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:35:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:35:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:35:45 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:35:50 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 14:35:55 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 14:36:00 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 14:36:05 Waiting for app: TestAllowInterBucketRecursion_function2 to get un-deployed
2021/12/04 14:36:05 App: TestAllowInterBucketRecursion_function2 got un-deployed
2021/12/04 14:36:05 No eventing-con process running
2021/12/04 14:36:05 Function: TestAllowInterBucketRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestAllowInterBucketRecursion_function2 deleting in the background"
}
2021/12/04 14:36:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:11 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:36:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestAllowInterBucketRecursion (115.15s)
=== RUN   TestInterBucketRecursion2
2021/12/04 14:36:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:36 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:36:47 Deploying app: TestInterBucketRecursion2_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:36:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 14:36:53 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 14:36:58 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 14:37:03 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 14:37:08 Waiting for app: TestInterBucketRecursion2_function1 to get deployed
2021/12/04 14:37:08 App: TestInterBucketRecursion2_function1 got deployed
2021/12/04 14:37:08 Deploying app: TestInterBucketRecursion2_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:37:10 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 14:37:15 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 14:37:20 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 14:37:25 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 14:37:30 Waiting for app: TestInterBucketRecursion2_function2 to get deployed
2021/12/04 14:37:30 App: TestInterBucketRecursion2_function2 got deployed
2021/12/04 14:37:30 Deploying app: TestInterBucketRecursion2_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:37:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestInterBucketRecursion2_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 14:37:36 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 14:37:41 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 14:37:46 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 14:37:51 Waiting for app: TestInterBucketRecursion2_function3 to get deployed
2021/12/04 14:37:51 App: TestInterBucketRecursion2_function3 got deployed
2021/12/04 14:37:51 Deploying app: TestInterBucketRecursion2_function4. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:37:51 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestInterBucketRecursion2_function4 causes a cycle involving functions: [TestInterBucketRecursion2_function1 TestInterBucketRecursion2_function2 TestInterBucketRecursion2_function3], hence deployment is disallowed"
 }
}
2021/12/04 14:37:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:37:56 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 14:38:01 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 14:38:06 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 14:38:11 Waiting for app: TestInterBucketRecursion2_function1 to get un-deployed
2021/12/04 14:38:11 App: TestInterBucketRecursion2_function1 got un-deployed
2021/12/04 14:38:11 No eventing-con process running
2021/12/04 14:38:11 Function: TestInterBucketRecursion2_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function1 deleting in the background"
}
2021/12/04 14:38:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:38:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:38:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:38:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:38:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:38:29 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 14:38:34 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 14:38:39 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 14:38:44 Waiting for app: TestInterBucketRecursion2_function2 to get un-deployed
2021/12/04 14:38:44 App: TestInterBucketRecursion2_function2 got un-deployed
2021/12/04 14:38:44 No eventing-con process running
2021/12/04 14:38:45 Function: TestInterBucketRecursion2_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function2 deleting in the background"
}
2021/12/04 14:38:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:38:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:38:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:38:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:38:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:39:03 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 14:39:08 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 14:39:13 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 14:39:18 Waiting for app: TestInterBucketRecursion2_function3 to get un-deployed
2021/12/04 14:39:18 App: TestInterBucketRecursion2_function3 got un-deployed
2021/12/04 14:39:18 No eventing-con process running
2021/12/04 14:39:18 Function: TestInterBucketRecursion2_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestInterBucketRecursion2_function3 deleting in the background"
}
2021/12/04 14:39:20 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:39:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:39:26 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:39:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:39:36 Waiting for app: TestInterBucketRecursion2_function4 to get un-deployed
2021/12/04 14:39:36 App: TestInterBucketRecursion2_function4 got un-deployed
2021/12/04 14:39:36 No eventing-con process running
2021/12/04 14:39:37 Function: TestInterBucketRecursion2_function4 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestInterBucketRecursion2_function4 not found"
 }
}
2021/12/04 14:39:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:39:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:39:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:39:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestInterBucketRecursion2 (213.82s)
=== RUN   TestBucketDeleteWithRebOut
2021/12/04 14:39:57 Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 14:40:02 Rebalance progress: 100

VB distribution across eventing nodes and their workers::

2021/12/04 14:40:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:40:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:40:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:40:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:41:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:41:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:41:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:41:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:42:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:42:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:42:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:42:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:43:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:43:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:43:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:43:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:44:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:44:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:44:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:44:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:45:05 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:45:20 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:45:35 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]
2021/12/04 14:45:50 waitForIndexes waiting for indexes: [{Name:#primary Bucket:eventing Completion:66} {Name:#primary Bucket:hello-world Completion:100}]

DCP Stream statuses:

vbucket curr owner:


2021/12/04 14:46:05 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:46:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 14:46:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 14:46:11 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/04 14:46:11 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:46:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 14:46:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 14:46:16 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:

vbucket curr owner:


2021/12/04 14:46:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:46:21 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/04 14:46:21 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/04 14:46:21 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:46:21 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestBucketDeleteWithRebOut not found"
 }
}
2021/12/04 14:46:22 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:46:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:46:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:46:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:46:39 eventingSetup.go:125: Deploying app: TestBucketDeleteWithRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:46:40 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBucketDeleteWithRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 14:46:45 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 14:46:50 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 14:46:55 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 14:47:00 clusterSetup.go:346: Waiting for app: TestBucketDeleteWithRebOut to get deployed
2021/12/04 14:47:00 clusterSetup.go:354: App: TestBucketDeleteWithRebOut got deployed

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: a47b3cd7b6fcba347c941e3c44f95133
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
Producer node: 172.31.2.134:9300	Node UUID: c6c72bf434ed0a3c7eb5d258bdbe00f0
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
Producer node: 127.0.0.1:9303	Node UUID: 29beee3f904fdf467b8a8a471bd9ecf8
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/04 14:47:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:47:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 14:47:05 Rebalance progress: 29.16666666666666
2021/12/04 14:47:10 Rebalance progress: 29.16666666666666
2021/12/04 14:47:15 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 1024
	vb list dump: [0-1023]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: a47b3cd7b6fcba347c941e3c44f95133
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [0-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [114-227]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 114
	v: [228-341]
Producer node: 172.31.2.134:9300	Node UUID: c6c72bf434ed0a3c7eb5d258bdbe00f0
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [683-796]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [797-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [911-1023]
Producer node: 127.0.0.1:9303	Node UUID: 29beee3f904fdf467b8a8a471bd9ecf8
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 114
	v: [342-455]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 114
	v: [456-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 113
	v: [570-682]


2021/12/04 14:47:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:47:15 handler_test.go:1674: Deleting source bucket: default
2021/12/04 14:47:17 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 14:47:18 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 14:47:23 Rebalance progress: 29.16666666666666
2021/12/04 14:47:28 Rebalance progress: 29.16666666666666
2021/12/04 14:47:33 Rebalance progress: 100

VB distribution across eventing nodes and their workers::


DCP Stream statuses:
	stream status: running
	len: 906
	vb list dump: [0-3, 5-6, 8-9, 11-26, 28-39, 41-43, 45, 47-54, 56-64, 66-74, 76-80, 82-89, 91-93, 95-104, 106-117, 121-125, 127-128, 130-132, 134-158, 160-173, 175-180, 182-198, 200-215, 217-219, 221-226, 228-235, 237-240, 242-250, 252-268, 270-271, 273-275, 278-280, 283-304, 306-310, 312-324, 326-331, 333-352, 354-367, 369-379, 381-391, 393-422, 424-425, 427-429, 431-434, 436-440, 442, 445-447, 449-453, 455-456, 458, 460-467, 469, 471-473, 475-476, 478-485, 488-492, 494, 496, 498-502, 504-505, 507-585, 588-596, 598-599, 601-607, 609-616, 618-620, 622-626, 628-631, 633-637, 639-641, 643-645, 647-682, 684-687, 690-707, 709-719, 721-723, 725-743, 745-754, 756, 758, 760-762, 764-771, 773-782, 784, 786-790, 792-804, 806-820, 822-831, 833-836, 838-843, 845-846, 848-849, 851-852, 854-862, 864-887, 889-890, 892-893, 895-902, 904-917, 919-921, 923, 925, 927-933, 935-942, 944-948, 950-965, 967-971, 973-992, 995-1017, 1019-1023]
	stream status: stopped
	len: 118
	vb list dump: [4, 7, 10, 27, 40, 44, 46, 55, 65, 75, 81, 90, 94, 105, 118-120, 126, 129, 133, 159, 174, 181, 199, 216, 220, 227, 236, 241, 251, 269, 272, 276-277, 281-282, 305, 311, 325, 332, 353, 368, 380, 392, 423, 426, 430, 435, 441, 443-444, 448, 454, 457, 459, 468, 470, 474, 477, 486-487, 493, 495, 497, 503, 506, 586-587, 597, 600, 608, 617, 621, 627, 632, 638, 642, 646, 683, 688-689, 708, 720, 724, 744, 755, 757, 759, 763, 772, 783, 785, 791, 805, 821, 832, 837, 844, 847, 850, 853, 863, 888, 891, 894, 903, 918, 922, 924, 926, 934, 943, 949, 966, 972, 993-994, 1018]

vbucket curr owner:
Producer node: 127.0.0.1:9302	Node UUID: a47b3cd7b6fcba347c941e3c44f95133
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 100
	v: [0-3, 5-6, 8-9, 11-26, 28-39, 41-43, 45, 47-54, 56-64, 66-74, 76-80, 82-89, 91-93, 95-104, 106-113]
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [114-117, 121-125, 127-128, 130-132, 134-158, 160-173, 175-180, 182-198, 200-215, 217-219, 221-226]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [228-235, 237-240, 242-250, 252-268, 270-271, 273-275, 278-280, 283-304, 306-310, 312-324, 326-331, 333-341]
Producer node: 172.31.2.134:9300	Node UUID: c6c72bf434ed0a3c7eb5d258bdbe00f0
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [797-804, 806-820, 822-831, 833-836, 838-843, 845-846, 848-849, 851-852, 854-862, 864-887, 889-890, 892-893, 895-902, 904-910]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [911-917, 919-921, 923, 925, 927-933, 935-942, 944-948, 950-965, 967-971, 973-992, 995-1017, 1019-1023]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 99
	v: [684-687, 690-707, 709-719, 721-723, 725-743, 745-754, 756, 758, 760-762, 764-771, 773-782, 784, 786-790, 792-796]
Producer node: 127.0.0.1:9303	Node UUID: 29beee3f904fdf467b8a8a471bd9ecf8
	workerID: worker_TestBucketDeleteWithRebOut_1
	len: 101
	v: [456, 458, 460-467, 469, 471-473, 475-476, 478-485, 488-492, 494, 496, 498-502, 504-505, 507-569]
	workerID: worker_TestBucketDeleteWithRebOut_2
	len: 101
	v: [570-585, 588-596, 598-599, 601-607, 609-616, 618-620, 622-626, 628-631, 633-637, 639-641, 643-645, 647-682]
	workerID: worker_TestBucketDeleteWithRebOut_0
	len: 101
	v: [342-352, 354-367, 369-379, 381-391, 393-422, 424-425, 427-429, 431-434, 436-440, 442, 445-447, 449-453, 455]


2021/12/04 14:47:33 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:47:38 eventingSetup.go:719: Waiting for function: TestBucketDeleteWithRebOut status to change to undeployed
2021/12/04 14:47:38 eventingSetup.go:727: Function: TestBucketDeleteWithRebOut status changed to undeployed
2021/12/04 14:47:43 clusterSetup.go:397: Waiting for app: TestBucketDeleteWithRebOut to get un-deployed
2021/12/04 14:47:43 clusterSetup.go:405: App: TestBucketDeleteWithRebOut got un-deployed
2021/12/04 14:47:43 eventingSetup.go:46: No eventing-con process running
2021/12/04 14:47:43 eventingSetup.go:418: Function: TestBucketDeleteWithRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBucketDeleteWithRebOut deleting in the background"
}
2021/12/04 14:47:44 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:47:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:47:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:47:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:47:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 14:47:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 14:48:01 Rebalance progress: 100
--- PASS: TestBucketDeleteWithRebOut (492.34s)
=== RUN   TestN1QLRecursion
2021/12/04 14:48:04 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 14:48:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 14:48:04 Removing node(s): [] from the cluster
2021/12/04 14:48:09 Rebalance progress: 100
2021/12/04 14:48:15 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:48:21 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:48:27 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:48:32 Deploying app: TestN1QLRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:48:33 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 14:48:33 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:48:38 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 14:48:43 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 14:48:48 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 14:48:53 Waiting for app: TestN1QLRecursion_function1 to get deployed
2021/12/04 14:48:53 App: TestN1QLRecursion_function1 got deployed
2021/12/04 14:48:53 TestN1QLRecursion_function1 is deployed
2021/12/04 14:48:53 Deploying app: TestN1QLRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:48:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 14:48:55 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:49:00 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 14:49:05 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 14:49:10 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 14:49:15 Waiting for app: TestN1QLRecursion_function3 to get deployed
2021/12/04 14:49:15 App: TestN1QLRecursion_function3 got deployed
2021/12/04 14:49:15 TestN1QLRecursion_function3 is deployed
2021/12/04 14:49:15 Deploying app: TestN1QLRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:49:15 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
}
2021/12/04 14:49:15 response body {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Inter bucket recursion error; function: TestN1QLRecursion_function2 causes a cycle involving functions: [TestN1QLRecursion_function3 TestN1QLRecursion_function1], hence deployment is disallowed"
 }
} err 
2021/12/04 14:49:15 Success: TestN1QLRecursion_function2 is not deployed
2021/12/04 14:49:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:49:20 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 14:49:25 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 14:49:30 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 14:49:35 Waiting for app: TestN1QLRecursion_function1 to get un-deployed
2021/12/04 14:49:35 App: TestN1QLRecursion_function1 got un-deployed
2021/12/04 14:49:35 No eventing-con process running
2021/12/04 14:49:35 Function: TestN1QLRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function1 deleting in the background"
}
2021/12/04 14:49:37 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:49:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:49:44 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:49:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:49:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:49:54 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 14:49:59 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 14:50:04 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 14:50:09 Waiting for app: TestN1QLRecursion_function3 to get un-deployed
2021/12/04 14:50:09 App: TestN1QLRecursion_function3 got un-deployed
2021/12/04 14:50:09 No eventing-con process running
2021/12/04 14:50:09 Function: TestN1QLRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLRecursion_function3 deleting in the background"
}
2021/12/04 14:50:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:50:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:50:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:50:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:50:24 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 14:50:29 Rebalance progress: 100
--- PASS: TestN1QLRecursion (145.14s)
=== RUN   TestN1QLAllowRecursion
2021/12/04 14:50:29 Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 14:50:29 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 14:50:29 Removing node(s): [] from the cluster
2021/12/04 14:50:34 Rebalance progress: 100
2021/12/04 14:50:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:50:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:50:53 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:50:58 Deploying app: TestN1QLAllowRecursion_function1. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:50:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
}
2021/12/04 14:50:59 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function1' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:51:04 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 14:51:09 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 14:51:14 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 14:51:19 Waiting for app: TestN1QLAllowRecursion_function1 to get deployed
2021/12/04 14:51:19 App: TestN1QLAllowRecursion_function1 got deployed
2021/12/04 14:51:19 TestN1QLAllowRecursion_function1 is deployed
2021/12/04 14:51:19 Deploying app: TestN1QLAllowRecursion_function3. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:51:20 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
}
2021/12/04 14:51:20 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function3' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:51:25 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 14:51:30 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 14:51:35 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 14:51:40 Waiting for app: TestN1QLAllowRecursion_function3 to get deployed
2021/12/04 14:51:40 App: TestN1QLAllowRecursion_function3 got deployed
2021/12/04 14:51:40 TestN1QLAllowRecursion_function3 is deployed
2021/12/04 14:51:40 Deploying app: TestN1QLAllowRecursion_function2. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:51:41 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
}
2021/12/04 14:51:41 response body {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAllowRecursion_function2' in metakv",
  "warnings": null
 }
} err 
2021/12/04 14:51:46 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 14:51:51 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 14:51:56 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 14:52:01 Waiting for app: TestN1QLAllowRecursion_function2 to get deployed
2021/12/04 14:52:01 App: TestN1QLAllowRecursion_function2 got deployed
2021/12/04 14:52:01 Success: TestN1QLAllowRecursion_function2 is deployed after allow_interbucket_recursion
2021/12/04 14:52:02 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:52:07 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 14:52:12 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 14:52:17 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 14:52:22 Waiting for app: TestN1QLAllowRecursion_function1 to get un-deployed
2021/12/04 14:52:22 App: TestN1QLAllowRecursion_function1 got un-deployed
2021/12/04 14:52:22 No eventing-con process running
2021/12/04 14:52:22 Function: TestN1QLAllowRecursion_function1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function1 deleting in the background"
}
2021/12/04 14:52:24 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:52:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:52:31 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:52:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:52:41 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 14:52:46 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 14:52:51 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 14:52:56 Waiting for app: TestN1QLAllowRecursion_function2 to get un-deployed
2021/12/04 14:52:56 App: TestN1QLAllowRecursion_function2 got un-deployed
2021/12/04 14:52:56 No eventing-con process running
2021/12/04 14:52:56 Function: TestN1QLAllowRecursion_function2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function2 deleting in the background"
}
2021/12/04 14:52:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:53:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:53:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:53:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:53:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:53:15 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 14:53:20 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 14:53:25 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 14:53:30 Waiting for app: TestN1QLAllowRecursion_function3 to get un-deployed
2021/12/04 14:53:30 App: TestN1QLAllowRecursion_function3 got un-deployed
2021/12/04 14:53:30 No eventing-con process running
2021/12/04 14:53:30 Function: TestN1QLAllowRecursion_function3 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAllowRecursion_function3 deleting in the background"
}
2021/12/04 14:53:32 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:53:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:53:39 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:53:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:53:45 Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 14:53:50 Rebalance progress: 100
--- PASS: TestN1QLAllowRecursion (200.97s)
=== RUN   TestOnDeleteExpiryBucketOp
2021/12/04 14:53:50 Sleeping for some time
2021/12/04 14:54:00 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 14:54:00 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/04 14:54:00 No eventing-con process running
2021/12/04 14:54:00 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestOnDeleteExpiryBucketOp not found"
 }
}
2021/12/04 14:54:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:54:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:54:09 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:54:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:54:14 Deploying app: TestOnDeleteExpiryBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:54:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestOnDeleteExpiryBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 14:54:15 Starting bucket ops to source bucket
2021/12/04 14:55:04 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 14:55:09 Waiting for dst bucket item count to get to: 5000 curr count: 1839
2021/12/04 14:55:14 Waiting for dst bucket item count to get to: 5000 curr count: 1839
2021/12/04 14:55:14 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 14:55:14 Starting bucket ops to source bucket
2021/12/04 14:55:15 Starting bucket ops to source bucket
2021/12/04 14:56:21 Waiting for dst bucket item count to get to: 7000 curr count: 5000
2021/12/04 14:56:21 src & dst bucket item count matched up. src bucket count: 7000 dst bucket count: 7000
2021/12/04 14:56:21 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:56:21 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:56:26 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 14:56:31 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 14:56:36 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 14:56:41 Waiting for app: TestOnDeleteExpiryBucketOp to get un-deployed
2021/12/04 14:56:41 App: TestOnDeleteExpiryBucketOp got un-deployed
2021/12/04 14:56:41 No eventing-con process running
2021/12/04 14:56:41 Function: TestOnDeleteExpiryBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestOnDeleteExpiryBucketOp deleting in the background"
}
2021/12/04 14:56:43 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:56:53 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 14:56:58 Waiting for dst bucket item count to get to: 0 curr count: 501
2021/12/04 14:56:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:57:00 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:57:10 Waiting for dst bucket item count to get to: 0 curr count: 4280
2021/12/04 14:57:15 Waiting for dst bucket item count to get to: 0 curr count: 4280
2021/12/04 14:57:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestOnDeleteExpiryBucketOp (204.81s)
=== RUN   TestTimerOverWriteSameReference
2021/12/04 14:57:20 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 14:57:20 App: TestTimerOverWriteSameReference got un-deployed
2021/12/04 14:57:20 No eventing-con process running
2021/12/04 14:57:20 Function: TestTimerOverWriteSameReference delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestTimerOverWriteSameReference not found"
 }
}
2021/12/04 14:57:22 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:57:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:57:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:57:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:57:34 Deploying app: TestTimerOverWriteSameReference. Item count src bucket: 0 dst bucket: 0
2021/12/04 14:57:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestTimerOverWriteSameReference' in metakv",
  "warnings": null
 }
}
2021/12/04 14:57:40 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 14:57:45 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 14:57:50 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 14:57:55 Waiting for app: TestTimerOverWriteSameReference to get deployed
2021/12/04 14:57:55 App: TestTimerOverWriteSameReference got deployed
2021/12/04 14:58:05 Starting bucket ops to source bucket
2021/12/04 14:59:06 src & dst bucket item count matched up. src bucket count: 1 dst bucket count: 1
2021/12/04 14:59:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 14:59:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 14:59:21 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 14:59:26 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 14:59:31 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 14:59:36 Waiting for app: TestTimerOverWriteSameReference to get un-deployed
2021/12/04 14:59:36 App: TestTimerOverWriteSameReference got un-deployed
2021/12/04 14:59:36 No eventing-con process running
2021/12/04 14:59:36 Function: TestTimerOverWriteSameReference delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestTimerOverWriteSameReference deleting in the background"
}
2021/12/04 14:59:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 14:59:48 Waiting for dst bucket item count to get to: 0 curr count: 654
2021/12/04 14:59:53 Waiting for dst bucket item count to get to: 0 curr count: 154
2021/12/04 14:59:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 14:59:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:00:05 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 15:00:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestTimerOverWriteSameReference (169.89s)
=== RUN   TestCancelTimerBucketop
2021/12/04 15:00:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 15:00:10 App: TestCancelTimerBucketop got un-deployed
2021/12/04 15:00:10 No eventing-con process running
2021/12/04 15:00:10 Function: TestCancelTimerBucketop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCancelTimerBucketop not found"
 }
}
2021/12/04 15:00:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:00:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:00:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:00:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:00:24 Deploying app: TestCancelTimerBucketop. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:00:25 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCancelTimerBucketop' in metakv",
  "warnings": null
 }
}
2021/12/04 15:00:30 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 15:00:35 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 15:00:40 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 15:00:45 Waiting for app: TestCancelTimerBucketop to get deployed
2021/12/04 15:00:45 App: TestCancelTimerBucketop got deployed
2021/12/04 15:00:45 Starting bucket ops to source bucket
2021/12/04 15:00:52 Starting bucket ops to source bucket
2021/12/04 15:01:54 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 15:01:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:01:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:02:00 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 15:02:05 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 15:02:10 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 15:02:15 Waiting for app: TestCancelTimerBucketop to get un-deployed
2021/12/04 15:02:15 App: TestCancelTimerBucketop got un-deployed
2021/12/04 15:02:15 No eventing-con process running
2021/12/04 15:02:15 Function: TestCancelTimerBucketop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCancelTimerBucketop deleting in the background"
}
2021/12/04 15:02:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:02:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:02:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:02:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:02:38 Waiting for dst bucket item count to get to: 0 curr count: 870
2021/12/04 15:02:43 Waiting for dst bucket item count to get to: 0 curr count: 370
2021/12/04 15:02:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCancelTimerBucketop (158.57s)
=== RUN   TestJSExpiryDate
2021/12/04 15:02:48 Starting bucket ops to source bucket
2021/12/04 15:03:06 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 15:03:06 Deploying app: TestJSExpiryDate. Item count src bucket: 5000 dst bucket: 0
2021/12/04 15:03:08 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestJSExpiryDate' in metakv",
  "warnings": null
 }
}
2021/12/04 15:03:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:03:33 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:03:33 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:03:38 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 15:03:43 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 15:03:48 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 15:03:53 Waiting for app: TestJSExpiryDate to get un-deployed
2021/12/04 15:03:53 App: TestJSExpiryDate got un-deployed
2021/12/04 15:03:53 No eventing-con process running
2021/12/04 15:03:53 Function: TestJSExpiryDate delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestJSExpiryDate deleting in the background"
}
2021/12/04 15:03:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:04:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:04:02 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:04:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestJSExpiryDate (83.79s)
=== RUN   TestBinaryDoc
2021/12/04 15:04:12 Starting bucket ops to source bucket
2021/12/04 15:04:12 Deploying app: TestBinaryDoc. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:04:14 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestBinaryDoc' in metakv",
  "warnings": null
 }
}
2021/12/04 15:04:14 Starting bucket ops to source bucket
2021/12/04 15:04:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:04:14 Deleting source bucket docs now
2021/12/04 15:04:14 Starting bucket ops to source bucket
2021/12/04 15:04:51 Waiting for dst bucket item count to get to: 32 curr count: 24
2021/12/04 15:04:56 Waiting for dst bucket item count to get to: 32 curr count: 28
2021/12/04 15:04:56 src & dst bucket item count matched up. src bucket count: 32 dst bucket count: 32
2021/12/04 15:04:56 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:04:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:05:01 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 15:05:06 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 15:05:11 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 15:05:16 Waiting for app: TestBinaryDoc to get un-deployed
2021/12/04 15:05:16 App: TestBinaryDoc got un-deployed
2021/12/04 15:05:16 No eventing-con process running
2021/12/04 15:05:16 Function: TestBinaryDoc delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestBinaryDoc deleting in the background"
}
2021/12/04 15:05:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:05:28 Waiting for dst bucket item count to get to: 0 curr count: 19
2021/12/04 15:05:33 Waiting for dst bucket item count to get to: 0 curr count: 1
2021/12/04 15:05:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:05:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:05:45 Waiting for dst bucket item count to get to: 0 curr count: 32
2021/12/04 15:05:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBinaryDoc (97.34s)
=== RUN   TestConstantBindings
2021/12/04 15:05:45 Deploying app: TestConstantBindings. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:05:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestConstantBindings' in metakv",
  "warnings": null
 }
}
2021/12/04 15:05:46 Starting bucket ops to source bucket
2021/12/04 15:05:51 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 15:05:56 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 15:06:01 Waiting for dst bucket item count to get to: 6000 curr count: 0
2021/12/04 15:06:06 Waiting for dst bucket item count to get to: 6000 curr count: 2560
2021/12/04 15:06:11 Waiting for dst bucket item count to get to: 6000 curr count: 2560
2021/12/04 15:06:11 src & dst bucket item count matched up. src bucket count: 6000 dst bucket count: 6000
2021/12/04 15:06:11 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:06:12 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:06:17 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 15:06:22 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 15:06:27 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 15:06:32 Waiting for app: TestConstantBindings to get un-deployed
2021/12/04 15:06:32 App: TestConstantBindings got un-deployed
2021/12/04 15:06:32 No eventing-con process running
2021/12/04 15:06:32 Function: TestConstantBindings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestConstantBindings deleting in the background"
}
2021/12/04 15:06:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:06:43 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:06:48 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 15:06:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:06:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:07:00 Waiting for dst bucket item count to get to: 0 curr count: 4508
2021/12/04 15:07:05 Waiting for dst bucket item count to get to: 0 curr count: 4508
2021/12/04 15:07:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestConstantBindings (80.73s)
=== RUN   TestCodeInsight
2021/12/04 15:07:10 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 15:07:10 App: TestCodeInsight got un-deployed
2021/12/04 15:07:10 No eventing-con process running
2021/12/04 15:07:10 Function: TestCodeInsight delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestCodeInsight not found"
 }
}
2021/12/04 15:07:12 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:07:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:07:19 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:07:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:07:24 Starting bucket ops to source bucket
2021/12/04 15:07:25 Deploying app: TestCodeInsight. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:07:26 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestCodeInsight' in metakv",
  "warnings": null
 }
}
2021/12/04 15:07:31 Waiting for app: TestCodeInsight to get deployed
2021/12/04 15:07:36 Waiting for app: TestCodeInsight to get deployed
2021/12/04 15:07:41 Waiting for app: TestCodeInsight to get deployed
2021/12/04 15:07:46 Waiting for app: TestCodeInsight to get deployed
2021/12/04 15:07:46 App: TestCodeInsight got deployed
2021/12/04 15:07:51 Waiting for dst bucket item count to get to: 1000 curr count: 567
2021/12/04 15:07:51 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 15:07:51 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:07:56 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 15:08:01 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 15:08:06 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 15:08:11 Waiting for app: TestCodeInsight to get un-deployed
2021/12/04 15:08:11 App: TestCodeInsight got un-deployed
2021/12/04 15:08:11 No eventing-con process running
2021/12/04 15:08:11 Function: TestCodeInsight delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestCodeInsight deleting in the background"
}
2021/12/04 15:08:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:08:23 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:08:28 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 15:08:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:08:30 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:08:40 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 15:08:45 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 15:08:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestCodeInsight (99.50s)
=== RUN   TestParserTransform
--- PASS: TestParserTransform (0.04s)
=== RUN   TestParserDetect
--- PASS: TestParserDetect (0.00s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12933.072s

Functional tests took 3 hours, 35 minutes and 38 seconds.

N1ql tests

2021/12/04 15:08:54 Started n1ql
2021/12/04 15:09:03 Adding node: https://127.0.0.1:19001 with role: kv,index,n1ql to the cluster
2021/12/04 15:09:04 Error in adding nodes : , response: ["Prepare join failed. Node is already part of cluster."]
2021/12/04 15:09:04 Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 15:09:04 Removing node(s): [] from the cluster
2021/12/04 15:09:09 Rebalance progress: 100
=== RUN   TestRecursiveMutationN1QL
2021/12/04 15:09:14 Waiting for app: TestRecursiveMutationN1QL to get un-deployed
2021/12/04 15:09:14 App: TestRecursiveMutationN1QL got un-deployed
2021/12/04 15:09:14 No eventing-con process running
2021/12/04 15:09:14 Function: TestRecursiveMutationN1QL delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestRecursiveMutationN1QL not found"
 }
}
2021/12/04 15:09:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:09:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:09:23 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:09:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:09:28 Deploying app: TestRecursiveMutationN1QL. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:09:28 Context: Post to main store, response code: 400 dump: {
 "name": "ERR_INTER_BUCKET_RECURSION",
 "code": 51,
 "description": "Inter bucket recursion error, deployment of current handler will cause inter bucket recursion",
 "attributes": null,
 "runtime_info": {
  "code": 51,
  "info": "Function: TestRecursiveMutationN1QL N1QL DML to source keyspace default:default"
 }
}
--- PASS: TestRecursiveMutationN1QL (18.65s)
=== RUN   TestFlexReset1
2021/12/04 15:09:33 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 15:09:33 App: TestFlexReset1 got un-deployed
2021/12/04 15:09:33 No eventing-con process running
2021/12/04 15:09:33 Function: TestFlexReset1 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset1 not found"
 }
}
2021/12/04 15:09:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:09:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:09:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:09:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:09:47 Starting bucket ops to source bucket
2021/12/04 15:09:47 Deploying app: TestFlexReset1. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:09:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset1' in metakv",
  "warnings": null
 }
}
2021/12/04 15:09:53 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 15:09:58 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 15:10:03 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 15:10:08 Waiting for app: TestFlexReset1 to get deployed
2021/12/04 15:10:08 App: TestFlexReset1 got deployed
2021/12/04 15:10:14 Waiting for dst bucket item count to get to: 1000 curr count: 0
2021/12/04 15:10:14 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 15:10:14 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:10:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:10:19 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 15:10:24 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 15:10:29 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 15:10:34 Waiting for app: TestFlexReset1 to get un-deployed
2021/12/04 15:10:34 App: TestFlexReset1 got un-deployed
2021/12/04 15:10:34 No eventing-con process running
2021/12/04 15:10:34 Function: TestFlexReset1 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset1 deleting in the background"
}
2021/12/04 15:10:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:10:45 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:10:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:10:47 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:10:57 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:10:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset1 (89.30s)
=== RUN   TestFlexReset2
2021/12/04 15:11:02 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 15:11:02 App: TestFlexReset2 got un-deployed
2021/12/04 15:11:02 No eventing-con process running
2021/12/04 15:11:02 Function: TestFlexReset2 delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestFlexReset2 not found"
 }
}
2021/12/04 15:11:04 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:11:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:11:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:11:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:11:16 Starting bucket ops to source bucket
2021/12/04 15:11:17 Deploying app: TestFlexReset2. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:11:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestFlexReset2' in metakv",
  "warnings": null
 }
}
2021/12/04 15:11:23 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 15:11:28 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 15:11:33 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 15:11:38 Waiting for app: TestFlexReset2 to get deployed
2021/12/04 15:11:38 App: TestFlexReset2 got deployed
2021/12/04 15:11:43 Waiting for dst bucket item count to get to: 1000 curr count: 5
2021/12/04 15:11:43 src & dst bucket item count matched up. src bucket count: 1000 dst bucket count: 1000
2021/12/04 15:11:43 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:11:43 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:11:48 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 15:11:53 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 15:11:58 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 15:12:03 Waiting for app: TestFlexReset2 to get un-deployed
2021/12/04 15:12:03 App: TestFlexReset2 got un-deployed
2021/12/04 15:12:03 No eventing-con process running
2021/12/04 15:12:03 Function: TestFlexReset2 delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestFlexReset2 deleting in the background"
}
2021/12/04 15:12:05 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:12:15 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:12:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:12:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:12:27 Waiting for dst bucket item count to get to: 0 curr count: 1000
2021/12/04 15:12:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestFlexReset2 (89.25s)
=== RUN   TestN1QLLabelledBreak
2021/12/04 15:12:32 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 15:12:32 App: TestN1QLLabelledBreak got un-deployed
2021/12/04 15:12:32 No eventing-con process running
2021/12/04 15:12:32 Function: TestN1QLLabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLLabelledBreak not found"
 }
}
2021/12/04 15:12:33 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:12:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:12:40 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:12:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:12:45 Deploying app: TestN1QLLabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:12:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLLabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/04 15:12:51 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 15:12:56 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 15:13:01 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 15:13:06 Waiting for app: TestN1QLLabelledBreak to get deployed
2021/12/04 15:13:06 App: TestN1QLLabelledBreak got deployed
2021/12/04 15:13:06 Starting bucket ops to source bucket
2021/12/04 15:13:16 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/04 15:13:21 Waiting for dst bucket item count to get to: 10000 curr count: 235
2021/12/04 15:13:26 Waiting for dst bucket item count to get to: 10000 curr count: 235
2021/12/04 15:13:31 Waiting for dst bucket item count to get to: 10000 curr count: 3572
2021/12/04 15:13:36 Waiting for dst bucket item count to get to: 10000 curr count: 3572
2021/12/04 15:13:41 Waiting for dst bucket item count to get to: 10000 curr count: 7611
2021/12/04 15:13:46 Waiting for dst bucket item count to get to: 10000 curr count: 7611
2021/12/04 15:13:46 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 15:13:46 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:13:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:13:51 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 15:13:56 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 15:14:01 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 15:14:06 Waiting for app: TestN1QLLabelledBreak to get un-deployed
2021/12/04 15:14:06 App: TestN1QLLabelledBreak got un-deployed
2021/12/04 15:14:06 No eventing-con process running
2021/12/04 15:14:06 Function: TestN1QLLabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLLabelledBreak deleting in the background"
}
2021/12/04 15:14:08 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:14:18 Waiting for dst bucket item count to get to: 0 curr count: 3483
2021/12/04 15:14:23 Waiting for dst bucket item count to get to: 0 curr count: 983
2021/12/04 15:14:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:14:25 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:14:35 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 15:14:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLLabelledBreak (128.25s)
=== RUN   TestN1QLUnlabelledBreak
2021/12/04 15:14:40 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 15:14:40 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/04 15:14:40 No eventing-con process running
2021/12/04 15:14:40 Function: TestN1QLUnlabelledBreak delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLUnlabelledBreak not found"
 }
}
2021/12/04 15:14:42 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:14:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:14:48 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:14:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:14:53 Deploying app: TestN1QLUnlabelledBreak. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:14:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLUnlabelledBreak' in metakv",
  "warnings": null
 }
}
2021/12/04 15:15:00 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 15:15:05 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 15:15:10 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 15:15:15 Waiting for app: TestN1QLUnlabelledBreak to get deployed
2021/12/04 15:15:15 App: TestN1QLUnlabelledBreak got deployed
2021/12/04 15:15:15 Starting bucket ops to source bucket
2021/12/04 15:15:24 Waiting for dst bucket item count to get to: 10000 curr count: 261
2021/12/04 15:15:29 Waiting for dst bucket item count to get to: 10000 curr count: 261
2021/12/04 15:15:34 Waiting for dst bucket item count to get to: 10000 curr count: 767
2021/12/04 15:15:39 Waiting for dst bucket item count to get to: 10000 curr count: 3217
2021/12/04 15:15:44 Waiting for dst bucket item count to get to: 10000 curr count: 6043
2021/12/04 15:15:49 Waiting for dst bucket item count to get to: 10000 curr count: 8332
2021/12/04 15:15:49 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 15:15:49 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:15:49 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:15:54 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 15:15:59 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 15:16:04 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 15:16:09 Waiting for app: TestN1QLUnlabelledBreak to get un-deployed
2021/12/04 15:16:09 App: TestN1QLUnlabelledBreak got un-deployed
2021/12/04 15:16:09 No eventing-con process running
2021/12/04 15:16:10 Function: TestN1QLUnlabelledBreak delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLUnlabelledBreak deleting in the background"
}
2021/12/04 15:16:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:16:21 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 15:16:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 15:16:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:16:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:16:38 Waiting for dst bucket item count to get to: 0 curr count: 7525
2021/12/04 15:16:43 Waiting for dst bucket item count to get to: 0 curr count: 2525
2021/12/04 15:16:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLUnlabelledBreak (128.38s)
=== RUN   TestN1QLThrowStatement
2021/12/04 15:16:48 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 15:16:48 App: TestN1QLThrowStatement got un-deployed
2021/12/04 15:16:48 No eventing-con process running
2021/12/04 15:16:48 Function: TestN1QLThrowStatement delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLThrowStatement not found"
 }
}
2021/12/04 15:16:50 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:16:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:16:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:17:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:17:02 Deploying app: TestN1QLThrowStatement. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:17:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLThrowStatement' in metakv",
  "warnings": null
 }
}
2021/12/04 15:17:08 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 15:17:13 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 15:17:18 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 15:17:23 Waiting for app: TestN1QLThrowStatement to get deployed
2021/12/04 15:17:23 App: TestN1QLThrowStatement got deployed
2021/12/04 15:17:23 Starting bucket ops to source bucket
2021/12/04 15:17:32 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/04 15:17:37 Waiting for dst bucket item count to get to: 10000 curr count: 0
2021/12/04 15:17:42 Waiting for dst bucket item count to get to: 10000 curr count: 940
2021/12/04 15:17:47 Waiting for dst bucket item count to get to: 10000 curr count: 2886
2021/12/04 15:17:52 Waiting for dst bucket item count to get to: 10000 curr count: 4902
2021/12/04 15:17:57 Waiting for dst bucket item count to get to: 10000 curr count: 6908
2021/12/04 15:18:02 Waiting for dst bucket item count to get to: 10000 curr count: 8905
2021/12/04 15:18:07 Waiting for dst bucket item count to get to: 10000 curr count: 9661
2021/12/04 15:18:07 src & dst bucket item count matched up. src bucket count: 10000 dst bucket count: 10000
2021/12/04 15:18:07 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:18:07 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:18:12 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 15:18:18 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 15:18:23 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 15:18:28 Waiting for app: TestN1QLThrowStatement to get un-deployed
2021/12/04 15:18:28 App: TestN1QLThrowStatement got un-deployed
2021/12/04 15:18:28 No eventing-con process running
2021/12/04 15:18:28 Function: TestN1QLThrowStatement delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLThrowStatement deleting in the background"
}
2021/12/04 15:18:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:18:39 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 15:18:44 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 15:18:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:18:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:18:56 Waiting for dst bucket item count to get to: 0 curr count: 10000
2021/12/04 15:18:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLThrowStatement (133.13s)
=== RUN   TestN1QLNestedForLoop
2021/12/04 15:19:01 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 15:19:01 App: TestN1QLNestedForLoop got un-deployed
2021/12/04 15:19:01 No eventing-con process running
2021/12/04 15:19:01 Function: TestN1QLNestedForLoop delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLNestedForLoop not found"
 }
}
2021/12/04 15:19:03 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:19:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:19:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:19:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:19:15 Deploying app: TestN1QLNestedForLoop. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:19:16 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLNestedForLoop' in metakv",
  "warnings": null
 }
}
2021/12/04 15:19:21 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 15:19:26 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 15:19:31 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 15:19:36 Waiting for app: TestN1QLNestedForLoop to get deployed
2021/12/04 15:19:36 App: TestN1QLNestedForLoop got deployed
2021/12/04 15:19:36 Starting bucket ops to source bucket
2021/12/04 15:19:46 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 15:19:51 Waiting for dst bucket item count to get to: 5000 curr count: 144
2021/12/04 15:19:56 Waiting for dst bucket item count to get to: 5000 curr count: 144
2021/12/04 15:20:01 Waiting for dst bucket item count to get to: 5000 curr count: 2155
2021/12/04 15:20:06 Waiting for dst bucket item count to get to: 5000 curr count: 2155
2021/12/04 15:20:11 Waiting for dst bucket item count to get to: 5000 curr count: 4526
2021/12/04 15:20:16 Waiting for dst bucket item count to get to: 5000 curr count: 4526
2021/12/04 15:20:16 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 15:20:16 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:20:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:20:21 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 15:20:26 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 15:20:31 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 15:20:36 Waiting for app: TestN1QLNestedForLoop to get un-deployed
2021/12/04 15:20:36 App: TestN1QLNestedForLoop got un-deployed
2021/12/04 15:20:36 No eventing-con process running
2021/12/04 15:20:36 Function: TestN1QLNestedForLoop delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLNestedForLoop deleting in the background"
}
2021/12/04 15:20:38 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:20:48 Waiting for dst bucket item count to get to: 0 curr count: 2898
2021/12/04 15:20:53 Waiting for dst bucket item count to get to: 0 curr count: 398
2021/12/04 15:20:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:20:55 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:21:05 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 15:21:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLNestedForLoop (128.24s)
=== RUN   TestN1QLAutoClose
2021/12/04 15:21:10 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 15:21:10 App: TestN1QLAutoClose got un-deployed
2021/12/04 15:21:10 No eventing-con process running
2021/12/04 15:21:10 Function: TestN1QLAutoClose delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLAutoClose not found"
 }
}
2021/12/04 15:21:11 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:21:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:21:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:21:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:21:23 Deploying app: TestN1QLAutoClose. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:21:24 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLAutoClose' in metakv",
  "warnings": null
 }
}
2021/12/04 15:21:29 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 15:21:34 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 15:21:39 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 15:21:44 Waiting for app: TestN1QLAutoClose to get deployed
2021/12/04 15:21:44 App: TestN1QLAutoClose got deployed
2021/12/04 15:21:44 Starting bucket ops to source bucket
2021/12/04 15:21:50 Waiting for dst bucket item count to get to: 500 curr count: 0
2021/12/04 15:21:55 Waiting for dst bucket item count to get to: 500 curr count: 4
2021/12/04 15:22:00 Waiting for dst bucket item count to get to: 500 curr count: 8
2021/12/04 15:22:05 Waiting for dst bucket item count to get to: 500 curr count: 17
2021/12/04 15:22:10 Waiting for dst bucket item count to get to: 500 curr count: 29
2021/12/04 15:22:15 Waiting for dst bucket item count to get to: 500 curr count: 39
2021/12/04 15:22:20 Waiting for dst bucket item count to get to: 500 curr count: 51
2021/12/04 15:22:25 Waiting for dst bucket item count to get to: 500 curr count: 62
2021/12/04 15:22:30 Waiting for dst bucket item count to get to: 500 curr count: 73
2021/12/04 15:22:35 Waiting for dst bucket item count to get to: 500 curr count: 94
2021/12/04 15:22:40 Waiting for dst bucket item count to get to: 500 curr count: 94
2021/12/04 15:22:45 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/04 15:22:50 Waiting for dst bucket item count to get to: 500 curr count: 116
2021/12/04 15:22:55 Waiting for dst bucket item count to get to: 500 curr count: 126
2021/12/04 15:23:00 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/04 15:23:05 Waiting for dst bucket item count to get to: 500 curr count: 137
2021/12/04 15:23:10 Waiting for dst bucket item count to get to: 500 curr count: 158
2021/12/04 15:23:15 Waiting for dst bucket item count to get to: 500 curr count: 158
2021/12/04 15:23:20 Waiting for dst bucket item count to get to: 500 curr count: 179
2021/12/04 15:23:25 Waiting for dst bucket item count to get to: 500 curr count: 179
2021/12/04 15:23:30 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/04 15:23:35 Waiting for dst bucket item count to get to: 500 curr count: 202
2021/12/04 15:23:40 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/04 15:23:45 Waiting for dst bucket item count to get to: 500 curr count: 223
2021/12/04 15:23:50 Waiting for dst bucket item count to get to: 500 curr count: 245
2021/12/04 15:23:55 Waiting for dst bucket item count to get to: 500 curr count: 245
2021/12/04 15:24:00 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/04 15:24:05 Waiting for dst bucket item count to get to: 500 curr count: 265
2021/12/04 15:24:10 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/04 15:24:15 Waiting for dst bucket item count to get to: 500 curr count: 285
2021/12/04 15:24:20 Waiting for dst bucket item count to get to: 500 curr count: 308
2021/12/04 15:24:25 Waiting for dst bucket item count to get to: 500 curr count: 308
2021/12/04 15:24:30 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/04 15:24:35 Waiting for dst bucket item count to get to: 500 curr count: 329
2021/12/04 15:24:40 Waiting for dst bucket item count to get to: 500 curr count: 349
2021/12/04 15:24:45 Waiting for dst bucket item count to get to: 500 curr count: 349
2021/12/04 15:24:50 Waiting for dst bucket item count to get to: 500 curr count: 369
2021/12/04 15:24:55 Waiting for dst bucket item count to get to: 500 curr count: 378
2021/12/04 15:25:00 Waiting for dst bucket item count to get to: 500 curr count: 389
2021/12/04 15:25:05 Waiting for dst bucket item count to get to: 500 curr count: 400
2021/12/04 15:25:10 Waiting for dst bucket item count to get to: 500 curr count: 409
2021/12/04 15:25:15 Waiting for dst bucket item count to get to: 500 curr count: 421
2021/12/04 15:25:20 Waiting for dst bucket item count to get to: 500 curr count: 430
2021/12/04 15:25:25 Waiting for dst bucket item count to get to: 500 curr count: 442
2021/12/04 15:25:30 Waiting for dst bucket item count to get to: 500 curr count: 452
2021/12/04 15:25:35 Waiting for dst bucket item count to get to: 500 curr count: 463
2021/12/04 15:25:40 Waiting for dst bucket item count to get to: 500 curr count: 474
2021/12/04 15:25:45 Waiting for dst bucket item count to get to: 500 curr count: 483
2021/12/04 15:25:50 Waiting for dst bucket item count to get to: 500 curr count: 493
2021/12/04 15:25:55 Waiting for dst bucket item count to get to: 500 curr count: 498
2021/12/04 15:25:55 src & dst bucket item count matched up. src bucket count: 500 dst bucket count: 500
2021/12/04 15:25:55 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:25:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:26:01 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 15:26:06 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 15:26:11 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 15:26:16 Waiting for app: TestN1QLAutoClose to get un-deployed
2021/12/04 15:26:16 App: TestN1QLAutoClose got un-deployed
2021/12/04 15:26:16 No eventing-con process running
2021/12/04 15:26:16 Function: TestN1QLAutoClose delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLAutoClose deleting in the background"
}
2021/12/04 15:26:17 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:26:27 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/04 15:26:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:26:29 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:26:39 Waiting for dst bucket item count to get to: 0 curr count: 500
2021/12/04 15:26:44 Waiting for dst bucket item count to get to: 0 curr count: 250
2021/12/04 15:26:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLAutoClose (339.72s)
=== RUN   TestN1QLPosParams
2021/12/04 15:26:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 15:26:49 App: TestN1QLPosParams got un-deployed
2021/12/04 15:26:49 No eventing-con process running
2021/12/04 15:26:49 Function: TestN1QLPosParams delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLPosParams not found"
 }
}
2021/12/04 15:26:51 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:26:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:26:58 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:27:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:27:03 Deploying app: TestN1QLPosParams. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:27:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLPosParams' in metakv",
  "warnings": null
 }
}
2021/12/04 15:27:09 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 15:27:14 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 15:27:19 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 15:27:24 Waiting for app: TestN1QLPosParams to get deployed
2021/12/04 15:27:24 App: TestN1QLPosParams got deployed
2021/12/04 15:27:24 Starting bucket ops to source bucket
2021/12/04 15:27:34 Waiting for dst bucket item count to get to: 5000 curr count: 486
2021/12/04 15:27:39 Waiting for dst bucket item count to get to: 5000 curr count: 1439
2021/12/04 15:27:44 Waiting for dst bucket item count to get to: 5000 curr count: 3453
2021/12/04 15:27:44 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 15:27:44 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:27:44 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:27:49 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 15:27:54 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 15:27:59 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 15:28:04 Waiting for app: TestN1QLPosParams to get un-deployed
2021/12/04 15:28:04 App: TestN1QLPosParams got un-deployed
2021/12/04 15:28:04 No eventing-con process running
2021/12/04 15:28:04 Function: TestN1QLPosParams delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLPosParams deleting in the background"
}
2021/12/04 15:28:06 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:28:16 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 15:28:16 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:28:18 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:28:28 Waiting for dst bucket item count to get to: 0 curr count: 2500
2021/12/04 15:28:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLPosParams (103.35s)
=== RUN   TestN1QLExhaustConnPool
2021/12/04 15:28:33 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 15:28:33 App: TestN1QLExhaustConnPool got un-deployed
2021/12/04 15:28:33 No eventing-con process running
2021/12/04 15:28:33 Function: TestN1QLExhaustConnPool delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLExhaustConnPool not found"
 }
}
2021/12/04 15:28:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:28:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:28:41 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:28:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:28:46 Deploying app: TestN1QLExhaustConnPool. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:28:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLExhaustConnPool' in metakv",
  "warnings": null
 }
}
2021/12/04 15:28:52 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 15:28:57 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 15:29:02 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 15:29:07 Waiting for app: TestN1QLExhaustConnPool to get deployed
2021/12/04 15:29:08 App: TestN1QLExhaustConnPool got deployed
2021/12/04 15:29:08 Starting bucket ops to source bucket
2021/12/04 15:29:13 Waiting for dst bucket item count to get to: 100 curr count: 0
2021/12/04 15:29:18 Waiting for dst bucket item count to get to: 100 curr count: 27
2021/12/04 15:29:23 Waiting for dst bucket item count to get to: 100 curr count: 27
2021/12/04 15:29:23 src & dst bucket item count matched up. src bucket count: 100 dst bucket count: 100
2021/12/04 15:29:23 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:29:23 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:29:28 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 15:29:33 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 15:29:38 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 15:29:43 Waiting for app: TestN1QLExhaustConnPool to get un-deployed
2021/12/04 15:29:43 App: TestN1QLExhaustConnPool got un-deployed
2021/12/04 15:29:43 No eventing-con process running
2021/12/04 15:29:43 Function: TestN1QLExhaustConnPool delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLExhaustConnPool deleting in the background"
}
2021/12/04 15:29:45 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:29:55 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 15:29:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:29:57 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:30:07 Waiting for dst bucket item count to get to: 0 curr count: 100
2021/12/04 15:30:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLExhaustConnPool (98.97s)
=== RUN   TestN1QLGraceSufficientTimeOut
2021/12/04 15:30:12 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 15:30:12 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/04 15:30:12 No eventing-con process running
2021/12/04 15:30:12 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestN1QLGraceSufficientTimeOut not found"
 }
}
2021/12/04 15:30:13 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:30:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:30:20 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:30:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:30:25 Starting bucket ops to source bucket
2021/12/04 15:30:43 Starting bucket ops to source bucket
2021/12/04 15:31:01 Deploying app: TestN1QLGraceSufficientTimeOut. Item count src bucket: 30000 dst bucket: 8381
2021/12/04 15:31:02 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestN1QLGraceSufficientTimeOut' in metakv",
  "warnings": null
 }
}
2021/12/04 15:31:07 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 15:31:12 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 15:31:17 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 15:31:22 Waiting for app: TestN1QLGraceSufficientTimeOut to get deployed
2021/12/04 15:31:22 App: TestN1QLGraceSufficientTimeOut got deployed
2021/12/04 15:31:22 Starting bucket ops to source bucket
2021/12/04 15:31:27 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:32 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:37 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:42 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:47 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:52 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:31:58 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:32:03 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:32:08 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:32:13 Waiting for dst bucket item count to get to: 30001 curr count: 30000
2021/12/04 15:32:13 src & dst bucket item count matched up. src bucket count: 30001 dst bucket count: 30001
2021/12/04 15:32:13 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:32:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:32:18 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 15:32:23 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 15:32:28 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 15:32:33 Waiting for app: TestN1QLGraceSufficientTimeOut to get un-deployed
2021/12/04 15:32:33 App: TestN1QLGraceSufficientTimeOut got un-deployed
2021/12/04 15:32:33 No eventing-con process running
2021/12/04 15:32:33 Function: TestN1QLGraceSufficientTimeOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestN1QLGraceSufficientTimeOut deleting in the background"
}
2021/12/04 15:32:35 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:32:45 Waiting for dst bucket item count to get to: 0 curr count: 30000
2021/12/04 15:32:45 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:32:46 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:32:56 Waiting for dst bucket item count to get to: 0 curr count: 30001
2021/12/04 15:32:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLGraceSufficientTimeOut (169.87s)
=== RUN   TestN1QLQueryConsistency
2021/12/04 15:33:01 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 15:33:01 App: n1qlquery_consistency got un-deployed
2021/12/04 15:33:01 No eventing-con process running
2021/12/04 15:33:01 Function: n1qlquery_consistency delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: n1qlquery_consistency not found"
 }
}
2021/12/04 15:33:03 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:33:08 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:33:10 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:33:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:33:15 Starting bucket ops to source bucket
2021/12/04 15:33:18 Deploying app: n1qlquery_consistency. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:33:19 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'n1qlquery_consistency' in metakv",
  "warnings": null
 }
}
2021/12/04 15:33:24 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 15:33:29 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 15:33:34 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 15:33:39 Waiting for app: n1qlquery_consistency to get deployed
2021/12/04 15:33:39 App: n1qlquery_consistency got deployed
2021/12/04 15:33:44 Waiting for dst bucket item count to get to: 5000 curr count: 0
2021/12/04 15:33:49 Waiting for dst bucket item count to get to: 5000 curr count: 4051
2021/12/04 15:33:54 Waiting for dst bucket item count to get to: 5000 curr count: 4051
2021/12/04 15:33:54 src & dst bucket item count matched up. src bucket count: 5000 dst bucket count: 5000
2021/12/04 15:33:54 http call resp: Get "http://127.0.0.1:9301/api/v1/stats": dial tcp 127.0.0.1:9301: connect: connection refused
2021/12/04 15:33:54 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:33:59 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 15:34:04 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 15:34:09 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 15:34:14 Waiting for app: n1qlquery_consistency to get un-deployed
2021/12/04 15:34:14 App: n1qlquery_consistency got un-deployed
2021/12/04 15:34:14 No eventing-con process running
2021/12/04 15:34:14 Function: n1qlquery_consistency delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: n1qlquery_consistency deleting in the background"
}
2021/12/04 15:34:16 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:34:26 Waiting for dst bucket item count to get to: 0 curr count: 5000
2021/12/04 15:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:34:28 Context: Bucket flush, response code: 200 dump: 
2021/12/04 15:34:38 Waiting for dst bucket item count to get to: 0 curr count: 3737
2021/12/04 15:34:43 Waiting for dst bucket item count to get to: 0 curr count: 1237
2021/12/04 15:34:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestN1QLQueryConsistency (106.66s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	1545.168s

N1ql tests took 0 hours, 25 minutes and 49 seconds.

Starting server
Starting server: attempt 1

Rebalance tests

2021/12/04 15:36:18 Started rebalance
=== RUN   TestDuplicateWithEventingRebIn
2021/12/04 15:36:36 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 15:36:36 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/04 15:36:36 No eventing-con process running
2021/12/04 15:36:36 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebIn not found"
 }
}
2021/12/04 15:36:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:36:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:36:44 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:36:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:36:49 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:36:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:36:54 Deploying app: TestDuplicateWithEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:36:55 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 15:37:00 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 15:37:05 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 15:37:10 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 15:37:15 Waiting for app: TestDuplicateWithEventingRebIn to get deployed
2021/12/04 15:37:15 App: TestDuplicateWithEventingRebIn got deployed
2021/12/04 15:37:15 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 15:37:15 Starting bucket ops to source bucket
2021/12/04 15:37:30 Removing node(s): [] from the cluster
2021/12/04 15:37:35 Rebalance progress: 25
2021/12/04 15:37:40 Rebalance progress: 25
2021/12/04 15:37:45 Rebalance progress: 25
2021/12/04 15:37:50 Rebalance progress: 25
2021/12/04 15:37:55 Rebalance progress: 75
2021/12/04 15:38:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:38:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:38:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 15:38:00 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:38:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:38:05 Rebalance progress: 25
2021/12/04 15:38:10 Rebalance progress: 25
2021/12/04 15:38:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:38:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:38:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:38:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 15:38:15 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:38:15 Waiting for src bucket item count to get to: 200000 curr count: 83334
2021/12/04 15:38:20 Waiting for src bucket item count to get to: 200000 curr count: 83334
2021/12/04 15:38:25 Waiting for src bucket item count to get to: 200000 curr count: 99580
2021/12/04 15:38:30 Waiting for src bucket item count to get to: 200000 curr count: 99580
2021/12/04 15:38:35 Waiting for src bucket item count to get to: 200000 curr count: 115681
2021/12/04 15:38:40 Waiting for src bucket item count to get to: 200000 curr count: 115681
2021/12/04 15:38:45 Waiting for src bucket item count to get to: 200000 curr count: 131769
2021/12/04 15:38:50 Waiting for src bucket item count to get to: 200000 curr count: 131769
2021/12/04 15:38:55 Waiting for src bucket item count to get to: 200000 curr count: 148130
2021/12/04 15:39:00 Waiting for src bucket item count to get to: 200000 curr count: 148130
2021/12/04 15:39:05 Waiting for src bucket item count to get to: 200000 curr count: 164195
2021/12/04 15:39:10 Waiting for src bucket item count to get to: 200000 curr count: 164195
2021/12/04 15:39:15 Waiting for src bucket item count to get to: 200000 curr count: 180376
2021/12/04 15:39:20 Waiting for src bucket item count to get to: 200000 curr count: 180376
2021/12/04 15:39:25 Waiting for src bucket item count to get to: 200000 curr count: 196603
2021/12/04 15:39:30 Waiting for src bucket item count to get to: 200000 curr count: 196603
2021/12/04 15:39:40 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:39:45 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:39:50 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:39:55 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:00 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:05 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:10 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:15 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:20 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:25 Waiting for dst bucket item count to get to: 200000 curr count: 200001
2021/12/04 15:40:25 Post SET ops. Expected item count: 200000 got 200001
2021/12/04 15:40:25 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:40:30 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 15:40:35 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 15:40:40 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 15:40:45 Waiting for app: TestDuplicateWithEventingRebIn to get un-deployed
2021/12/04 15:40:45 App: TestDuplicateWithEventingRebIn got un-deployed
2021/12/04 15:40:45 No eventing-con process running
2021/12/04 15:40:45 Function: TestDuplicateWithEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebIn deleting in the background"
}
2021/12/04 15:40:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:40:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:40:57 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:40:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:40:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:41:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:41:09 Waiting for dst bucket item count to get to: 0 curr count: 200001
2021/12/04 15:41:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebIn (282.70s)
=== RUN   TestDuplicateWithEventingRebOut
2021/12/04 15:41:14 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 15:41:14 Removing node(s): [] from the cluster
2021/12/04 15:41:19 Rebalance progress: 100
2021/12/04 15:41:24 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 15:41:24 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/04 15:41:24 No eventing-con process running
2021/12/04 15:41:24 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingRebOut not found"
 }
}
2021/12/04 15:41:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:41:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:41:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:41:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:41:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:41:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:41:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:41:42 Deploying app: TestDuplicateWithEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:41:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 15:41:49 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 15:41:54 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 15:41:59 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 15:42:04 Waiting for app: TestDuplicateWithEventingRebOut to get deployed
2021/12/04 15:42:04 App: TestDuplicateWithEventingRebOut got deployed
2021/12/04 15:42:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:42:04 Starting bucket ops to source bucket
2021/12/04 15:42:09 Rebalance progress: 25
2021/12/04 15:42:14 Rebalance progress: 25
2021/12/04 15:42:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:42:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:42:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:42:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 15:42:19 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:42:19 Waiting for src bucket item count to get to: 200000 curr count: 8578
2021/12/04 15:42:24 Waiting for src bucket item count to get to: 200000 curr count: 24600
2021/12/04 15:42:29 Waiting for src bucket item count to get to: 200000 curr count: 24600
2021/12/04 15:42:34 Waiting for src bucket item count to get to: 200000 curr count: 40767
2021/12/04 15:42:39 Waiting for src bucket item count to get to: 200000 curr count: 40767
2021/12/04 15:42:44 Waiting for src bucket item count to get to: 200000 curr count: 57009
2021/12/04 15:42:49 Waiting for src bucket item count to get to: 200000 curr count: 57009
2021/12/04 15:42:54 Waiting for src bucket item count to get to: 200000 curr count: 57009
2021/12/04 15:42:59 Waiting for src bucket item count to get to: 200000 curr count: 73099
2021/12/04 15:43:04 Waiting for src bucket item count to get to: 200000 curr count: 73099
2021/12/04 15:43:09 Waiting for src bucket item count to get to: 200000 curr count: 89355
2021/12/04 15:43:14 Waiting for src bucket item count to get to: 200000 curr count: 89355
2021/12/04 15:43:19 Waiting for src bucket item count to get to: 200000 curr count: 105738
2021/12/04 15:43:24 Waiting for src bucket item count to get to: 200000 curr count: 105738
2021/12/04 15:43:29 Waiting for src bucket item count to get to: 200000 curr count: 122097
2021/12/04 15:43:34 Waiting for src bucket item count to get to: 200000 curr count: 122097
2021/12/04 15:43:39 Waiting for src bucket item count to get to: 200000 curr count: 138268
2021/12/04 15:43:44 Waiting for src bucket item count to get to: 200000 curr count: 138268
2021/12/04 15:43:49 Waiting for src bucket item count to get to: 200000 curr count: 154353
2021/12/04 15:43:54 Waiting for src bucket item count to get to: 200000 curr count: 170814
2021/12/04 15:43:59 Waiting for src bucket item count to get to: 200000 curr count: 170814
2021/12/04 15:44:04 Waiting for src bucket item count to get to: 200000 curr count: 186925
2021/12/04 15:44:09 Waiting for src bucket item count to get to: 200000 curr count: 186925
2021/12/04 15:44:14 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 15:44:14 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 15:44:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:44:19 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 15:44:24 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 15:44:29 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 15:44:34 Waiting for app: TestDuplicateWithEventingRebOut to get un-deployed
2021/12/04 15:44:34 App: TestDuplicateWithEventingRebOut got un-deployed
2021/12/04 15:44:34 No eventing-con process running
2021/12/04 15:44:34 Function: TestDuplicateWithEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingRebOut deleting in the background"
}
2021/12/04 15:44:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:44:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:44:46 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:44:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:44:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:44:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:44:58 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:44:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingRebOut (228.97s)
=== RUN   TestDuplicateWithEventingSwapReb
2021/12/04 15:45:08 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 15:45:08 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/04 15:45:08 No eventing-con process running
2021/12/04 15:45:08 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithEventingSwapReb not found"
 }
}
2021/12/04 15:45:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:45:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:45:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:45:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:45:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:45:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:45:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:45:26 Deploying app: TestDuplicateWithEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:45:27 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 15:45:32 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 15:45:37 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 15:45:42 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 15:45:47 Waiting for app: TestDuplicateWithEventingSwapReb to get deployed
2021/12/04 15:45:47 App: TestDuplicateWithEventingSwapReb got deployed
2021/12/04 15:45:47 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 15:45:47 Starting bucket ops to source bucket
2021/12/04 15:45:47 Removing node(s): [] from the cluster
2021/12/04 15:45:52 Rebalance progress: 25
2021/12/04 15:45:57 Rebalance progress: 25
2021/12/04 15:46:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:46:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9302/api/v1/stats": dial tcp 127.0.0.1:9302: connect: connection refused
2021/12/04 15:46:02 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:46:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 15:46:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:46:14 Rebalance progress: 16.66666666666667
2021/12/04 15:46:19 Rebalance progress: 16.66666666666667
2021/12/04 15:46:24 Rebalance progress: 16.66666666666667
2021/12/04 15:46:29 Rebalance progress: 72.22222222222223
2021/12/04 15:46:34 Rebalance progress: 72.22222222222223
2021/12/04 15:46:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:46:39 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:46:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 15:46:44 Rebalance progress: 16.66666666666667
2021/12/04 15:46:49 Rebalance progress: 16.66666666666667
2021/12/04 15:46:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:46:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:46:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:46:54 Waiting for src bucket item count to get to: 200000 curr count: 92039
2021/12/04 15:47:00 Waiting for src bucket item count to get to: 200000 curr count: 92039
2021/12/04 15:47:05 Waiting for src bucket item count to get to: 200000 curr count: 108049
2021/12/04 15:47:10 Waiting for src bucket item count to get to: 200000 curr count: 108049
2021/12/04 15:47:15 Waiting for src bucket item count to get to: 200000 curr count: 123963
2021/12/04 15:47:20 Waiting for src bucket item count to get to: 200000 curr count: 123963
2021/12/04 15:47:25 Waiting for src bucket item count to get to: 200000 curr count: 139741
2021/12/04 15:47:30 Waiting for src bucket item count to get to: 200000 curr count: 139741
2021/12/04 15:47:35 Waiting for src bucket item count to get to: 200000 curr count: 155814
2021/12/04 15:47:40 Waiting for src bucket item count to get to: 200000 curr count: 155814
2021/12/04 15:47:45 Waiting for src bucket item count to get to: 200000 curr count: 171513
2021/12/04 15:47:50 Waiting for src bucket item count to get to: 200000 curr count: 171513
2021/12/04 15:47:55 Waiting for src bucket item count to get to: 200000 curr count: 187573
2021/12/04 15:48:00 Waiting for src bucket item count to get to: 200000 curr count: 187573
2021/12/04 15:48:05 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 15:48:05 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 15:48:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:48:10 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 15:48:15 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 15:48:20 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 15:48:25 Waiting for app: TestDuplicateWithEventingSwapReb to get un-deployed
2021/12/04 15:48:25 App: TestDuplicateWithEventingSwapReb got un-deployed
2021/12/04 15:48:25 No eventing-con process running
2021/12/04 15:48:25 Function: TestDuplicateWithEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithEventingSwapReb deleting in the background"
}
2021/12/04 15:48:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:48:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:48:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:48:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:48:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:48:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:48:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:48:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithEventingSwapReb (230.78s)
=== RUN   TestDuplicateWithKVRebIn
2021/12/04 15:48:58 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 15:48:58 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 15:48:58 No eventing-con process running
2021/12/04 15:48:58 Function: TestDuplicateWithKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebIn not found"
 }
}
2021/12/04 15:49:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:49:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:49:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:49:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:49:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:49:12 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:49:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:49:17 Deploying app: TestDuplicateWithKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:49:18 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 15:49:23 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 15:49:28 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 15:49:33 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 15:49:38 Waiting for app: TestDuplicateWithKVRebIn to get deployed
2021/12/04 15:49:38 App: TestDuplicateWithKVRebIn got deployed
2021/12/04 15:49:38 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 15:49:38 Starting bucket ops to source bucket
2021/12/04 15:49:38 Removing node(s): [] from the cluster
2021/12/04 15:49:44 Rebalance progress: 16.66666666666667
2021/12/04 15:49:49 Rebalance progress: 16.66666666666667
2021/12/04 15:49:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:49:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:49:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:49:54 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:49:54 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:49:59 Rebalance progress: 16.66666666666667
2021/12/04 15:50:04 Rebalance progress: 16.66666666666667
2021/12/04 15:50:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:50:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:50:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:50:09 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:50:09 Waiting for src bucket item count to get to: 200000 curr count: 32145
2021/12/04 15:50:14 Waiting for src bucket item count to get to: 200000 curr count: 48164
2021/12/04 15:50:19 Waiting for src bucket item count to get to: 200000 curr count: 48164
2021/12/04 15:50:24 Waiting for src bucket item count to get to: 200000 curr count: 64491
2021/12/04 15:50:29 Waiting for src bucket item count to get to: 200000 curr count: 64491
2021/12/04 15:50:34 Waiting for src bucket item count to get to: 200000 curr count: 79757
2021/12/04 15:50:39 Waiting for src bucket item count to get to: 200000 curr count: 79757
2021/12/04 15:50:44 Waiting for src bucket item count to get to: 200000 curr count: 95417
2021/12/04 15:50:49 Waiting for src bucket item count to get to: 200000 curr count: 95417
2021/12/04 15:50:54 Waiting for src bucket item count to get to: 200000 curr count: 111233
2021/12/04 15:50:59 Waiting for src bucket item count to get to: 200000 curr count: 111233
2021/12/04 15:51:04 Waiting for src bucket item count to get to: 200000 curr count: 111233
2021/12/04 15:51:09 Waiting for src bucket item count to get to: 200000 curr count: 127364
2021/12/04 15:51:14 Waiting for src bucket item count to get to: 200000 curr count: 127364
2021/12/04 15:51:19 Waiting for src bucket item count to get to: 200000 curr count: 143688
2021/12/04 15:51:24 Waiting for src bucket item count to get to: 200000 curr count: 143688
2021/12/04 15:51:29 Waiting for src bucket item count to get to: 200000 curr count: 159828
2021/12/04 15:51:34 Waiting for src bucket item count to get to: 200000 curr count: 159828
2021/12/04 15:51:39 Waiting for src bucket item count to get to: 200000 curr count: 175106
2021/12/04 15:51:44 Waiting for src bucket item count to get to: 200000 curr count: 175106
2021/12/04 15:51:49 Waiting for src bucket item count to get to: 200000 curr count: 190960
2021/12/04 15:51:54 Waiting for src bucket item count to get to: 200000 curr count: 190960
2021/12/04 15:51:59 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 15:51:59 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 15:51:59 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:52:04 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 15:52:09 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 15:52:14 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 15:52:19 Waiting for app: TestDuplicateWithKVRebIn to get un-deployed
2021/12/04 15:52:19 App: TestDuplicateWithKVRebIn got un-deployed
2021/12/04 15:52:19 No eventing-con process running
2021/12/04 15:52:19 Function: TestDuplicateWithKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebIn deleting in the background"
}
2021/12/04 15:52:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:52:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:52:31 Waiting for dst bucket item count to get to: 0 curr count: 156786
2021/12/04 15:52:36 Waiting for dst bucket item count to get to: 0 curr count: 156786
2021/12/04 15:52:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:52:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:52:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:52:48 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:52:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebIn (239.32s)
=== RUN   TestDuplicateWithKVRebOut
2021/12/04 15:52:53 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 15:52:53 Removing node(s): [] from the cluster
2021/12/04 15:52:58 Rebalance progress: 100
2021/12/04 15:53:03 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 15:53:03 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 15:53:03 No eventing-con process running
2021/12/04 15:53:03 Function: TestDuplicateWithKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVRebOut not found"
 }
}
2021/12/04 15:53:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:53:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:53:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:53:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:53:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:53:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:53:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:53:22 Deploying app: TestDuplicateWithKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:53:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 15:53:28 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 15:53:33 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 15:53:38 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 15:53:43 Waiting for app: TestDuplicateWithKVRebOut to get deployed
2021/12/04 15:53:43 App: TestDuplicateWithKVRebOut got deployed
2021/12/04 15:53:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:53:43 Starting bucket ops to source bucket
2021/12/04 15:53:48 Rebalance progress: 16.66666666666667
2021/12/04 15:53:53 Rebalance progress: 16.66666666666667
2021/12/04 15:53:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:53:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:53:58 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:53:58 Waiting for src bucket item count to get to: 200000 curr count: 9621
2021/12/04 15:54:03 Waiting for src bucket item count to get to: 200000 curr count: 25710
2021/12/04 15:54:08 Waiting for src bucket item count to get to: 200000 curr count: 25710
2021/12/04 15:54:13 Waiting for src bucket item count to get to: 200000 curr count: 41767
2021/12/04 15:54:18 Waiting for src bucket item count to get to: 200000 curr count: 41767
2021/12/04 15:54:23 Waiting for src bucket item count to get to: 200000 curr count: 57975
2021/12/04 15:54:28 Waiting for src bucket item count to get to: 200000 curr count: 57975
2021/12/04 15:54:33 Waiting for src bucket item count to get to: 200000 curr count: 57975
2021/12/04 15:54:38 Waiting for src bucket item count to get to: 200000 curr count: 73347
2021/12/04 15:54:43 Waiting for src bucket item count to get to: 200000 curr count: 73347
2021/12/04 15:54:48 Waiting for src bucket item count to get to: 200000 curr count: 89382
2021/12/04 15:54:53 Waiting for src bucket item count to get to: 200000 curr count: 89382
2021/12/04 15:54:58 Waiting for src bucket item count to get to: 200000 curr count: 105471
2021/12/04 15:55:03 Waiting for src bucket item count to get to: 200000 curr count: 105471
2021/12/04 15:55:08 Waiting for src bucket item count to get to: 200000 curr count: 121559
2021/12/04 15:55:13 Waiting for src bucket item count to get to: 200000 curr count: 121559
2021/12/04 15:55:18 Waiting for src bucket item count to get to: 200000 curr count: 137680
2021/12/04 15:55:23 Waiting for src bucket item count to get to: 200000 curr count: 137680
2021/12/04 15:55:28 Waiting for src bucket item count to get to: 200000 curr count: 153781
2021/12/04 15:55:33 Waiting for src bucket item count to get to: 200000 curr count: 153781
2021/12/04 15:55:38 Waiting for src bucket item count to get to: 200000 curr count: 168957
2021/12/04 15:55:43 Waiting for src bucket item count to get to: 200000 curr count: 168957
2021/12/04 15:55:48 Waiting for src bucket item count to get to: 200000 curr count: 184715
2021/12/04 15:55:53 Waiting for src bucket item count to get to: 200000 curr count: 184715
2021/12/04 15:55:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 15:55:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 15:55:58 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:56:03 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 15:56:08 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 15:56:13 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 15:56:18 Waiting for app: TestDuplicateWithKVRebOut to get un-deployed
2021/12/04 15:56:18 App: TestDuplicateWithKVRebOut got un-deployed
2021/12/04 15:56:18 No eventing-con process running
2021/12/04 15:56:18 Function: TestDuplicateWithKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVRebOut deleting in the background"
}
2021/12/04 15:56:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:56:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:56:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:56:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:56:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:56:37 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 15:56:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVRebOut (229.10s)
=== RUN   TestDuplicateWithKVSwapReb
2021/12/04 15:56:47 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 15:56:47 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 15:56:47 No eventing-con process running
2021/12/04 15:56:47 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateWithKVSwapReb not found"
 }
}
2021/12/04 15:56:49 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:56:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:56:54 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 15:56:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:57:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 15:57:05 Deploying app: TestDuplicateWithKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 15:57:06 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateWithKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 15:57:11 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 15:57:16 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 15:57:21 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 15:57:26 Waiting for app: TestDuplicateWithKVSwapReb to get deployed
2021/12/04 15:57:26 App: TestDuplicateWithKVSwapReb got deployed
2021/12/04 15:57:26 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 15:57:26 Starting bucket ops to source bucket
2021/12/04 15:57:27 Removing node(s): [] from the cluster
2021/12/04 15:57:32 Rebalance progress: 16.66666666666667
2021/12/04 15:57:37 Rebalance progress: 16.66666666666667
2021/12/04 15:57:42 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:57:42 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:57:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 15:57:42 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 15:57:47 Rebalance progress: 16.66666666666667
2021/12/04 15:57:52 Rebalance progress: 16.66666666666667
2021/12/04 15:57:57 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:57:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:57:57 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:57:57 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 15:58:02 Rebalance progress: 16.66666666666667
2021/12/04 15:58:07 Rebalance progress: 16.66666666666667
2021/12/04 15:58:12 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 15:58:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 15:58:12 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 15:58:12 Waiting for src bucket item count to get to: 200000 curr count: 65222
2021/12/04 15:58:17 Waiting for src bucket item count to get to: 200000 curr count: 65222
2021/12/04 15:58:22 Waiting for src bucket item count to get to: 200000 curr count: 79909
2021/12/04 15:58:27 Waiting for src bucket item count to get to: 200000 curr count: 79909
2021/12/04 15:58:32 Waiting for src bucket item count to get to: 200000 curr count: 95429
2021/12/04 15:58:37 Waiting for src bucket item count to get to: 200000 curr count: 95429
2021/12/04 15:58:42 Waiting for src bucket item count to get to: 200000 curr count: 111301
2021/12/04 15:58:47 Waiting for src bucket item count to get to: 200000 curr count: 111301
2021/12/04 15:58:53 Waiting for src bucket item count to get to: 200000 curr count: 127041
2021/12/04 15:58:58 Waiting for src bucket item count to get to: 200000 curr count: 127041
2021/12/04 15:59:03 Waiting for src bucket item count to get to: 200000 curr count: 142826
2021/12/04 15:59:08 Waiting for src bucket item count to get to: 200000 curr count: 142826
2021/12/04 15:59:13 Waiting for src bucket item count to get to: 200000 curr count: 142826
2021/12/04 15:59:18 Waiting for src bucket item count to get to: 200000 curr count: 158469
2021/12/04 15:59:23 Waiting for src bucket item count to get to: 200000 curr count: 158469
2021/12/04 15:59:28 Waiting for src bucket item count to get to: 200000 curr count: 173379
2021/12/04 15:59:33 Waiting for src bucket item count to get to: 200000 curr count: 173379
2021/12/04 15:59:38 Waiting for src bucket item count to get to: 200000 curr count: 188803
2021/12/04 15:59:43 Waiting for src bucket item count to get to: 200000 curr count: 188803
2021/12/04 15:59:48 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 15:59:48 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 15:59:48 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 15:59:53 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 15:59:58 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 16:00:03 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 16:00:08 Waiting for app: TestDuplicateWithKVSwapReb to get un-deployed
2021/12/04 16:00:08 App: TestDuplicateWithKVSwapReb got un-deployed
2021/12/04 16:00:08 No eventing-con process running
2021/12/04 16:00:08 Function: TestDuplicateWithKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateWithKVSwapReb deleting in the background"
}
2021/12/04 16:00:10 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:00:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:00:15 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:00:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:00:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:00:26 Waiting for dst bucket item count to get to: 0 curr count: 200000
2021/12/04 16:00:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateWithKVSwapReb (229.61s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpOneByOne
2021/12/04 16:00:36 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:00:36 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:00:36 No eventing-con process running
2021/12/04 16:00:36 Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 16:00:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:00:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:00:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:00:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:00:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:00:50 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:00:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:00:55 Deploying app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:00:56 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 16:01:01 Starting bucket ops to source bucket
2021/12/04 16:01:06 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:01:11 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:01:16 Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:01:16 App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:01:16 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 16:01:16 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:01:16 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:01:21 Rebalance progress: 16.66666666666667
2021/12/04 16:01:26 Rebalance progress: 16.66666666666667
2021/12/04 16:01:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:01:31 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 16:01:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:01:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:01:37 Rebalance progress: 16.66666666666667
2021/12/04 16:01:42 Rebalance progress: 16.66666666666667
2021/12/04 16:01:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:01:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:01:47 eventingSetup.go:610: http call resp: Get "http://127.0.0.1:9303/api/v1/stats": dial tcp 127.0.0.1:9303: connect: connection refused
2021/12/04 16:01:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 16:01:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:01:57 Rebalance progress: 12.5
2021/12/04 16:02:02 Rebalance progress: 12.5
2021/12/04 16:02:07 Rebalance progress: 12.5
2021/12/04 16:02:12 Rebalance progress: 70.83333333333334
2021/12/04 16:02:17 Rebalance progress: 70.83333333333334
2021/12/04 16:02:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:02:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:02:27 Rebalance progress: 12.5
2021/12/04 16:02:32 Rebalance progress: 12.5
2021/12/04 16:02:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:02:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:02:42 Rebalance progress: 12.5
2021/12/04 16:02:47 Rebalance progress: 12.5
2021/12/04 16:02:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:02:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:02:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 16:02:57 Rebalance progress: 12.5
2021/12/04 16:03:02 Rebalance progress: 12.5
2021/12/04 16:03:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:03:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:03:12 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:03:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:03:23 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:03:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:03:28 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:03:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:03:28 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 16:03:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:03:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:03:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:03:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:03:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:03:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12628
2021/12/04 16:03:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpOneByOne (199.85s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce
2021/12/04 16:03:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 16:03:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 16:03:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:03:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce not found"
 }
}
2021/12/04 16:03:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:04:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:04:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:04:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:04:15 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:04:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce' in metakv",
  "warnings": null
 }
}
2021/12/04 16:04:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:04:26 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 16:04:31 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 16:04:36 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get deployed
2021/12/04 16:04:36 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:04:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:04:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:04:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 16:04:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:04:42 Rebalance progress: 12.5
2021/12/04 16:04:47 Rebalance progress: 12.5
2021/12/04 16:04:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:04:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:04:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 16:04:57 Rebalance progress: 12.5
2021/12/04 16:05:02 Rebalance progress: 12.5
2021/12/04 16:05:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:05:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:05:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:05:07 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:05:12 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 16:05:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 16:05:23 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 16:05:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce to get un-deployed
2021/12/04 16:05:28 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce got un-deployed
2021/12/04 16:05:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:05:28 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce deleting in the background"
}
2021/12/04 16:05:29 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:05:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:05:34 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:05:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:05:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:05:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4630
2021/12/04 16:05:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpAllAtOnce (120.00s)
=== RUN   TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings
2021/12/04 16:05:56 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 16:05:56 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 16:05:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:05:56 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings not found"
 }
}
2021/12/04 16:05:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:06:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:06:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:06:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:06:15 eventingSetup.go:125: Deploying app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:06:16 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings' in metakv",
  "warnings": null
 }
}
2021/12/04 16:06:21 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:06:26 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 16:06:31 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 16:06:36 clusterSetup.go:346: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get deployed
2021/12/04 16:06:36 clusterSetup.go:354: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:06:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:06:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:06:42 Rebalance progress: 12.5
2021/12/04 16:06:47 Rebalance progress: 12.5
2021/12/04 16:06:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:06:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:06:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:06:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:06:57 Rebalance progress: 12.5
2021/12/04 16:07:02 Rebalance progress: 12.5
2021/12/04 16:07:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:07:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 16:07:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:07:12 Rebalance progress: 12.5
2021/12/04 16:07:17 Rebalance progress: 12.5
2021/12/04 16:07:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:07:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:07:27 Rebalance progress: 12.5
2021/12/04 16:07:32 Rebalance progress: 12.5
2021/12/04 16:07:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:07:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:07:42 Rebalance progress: 12.5
2021/12/04 16:07:47 Rebalance progress: 12.5
2021/12/04 16:07:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:07:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:07:52 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 16:07:57 Rebalance progress: 12.5
2021/12/04 16:08:02 Rebalance progress: 12.5
2021/12/04 16:08:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:08:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:08:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:08:08 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:08:13 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 16:08:18 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 16:08:23 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 16:08:28 clusterSetup.go:397: Waiting for app: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings to get un-deployed
2021/12/04 16:08:28 clusterSetup.go:405: App: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings got un-deployed
2021/12/04 16:08:28 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:08:28 eventingSetup.go:418: Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings deleting in the background"
}
2021/12/04 16:08:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:08:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:08:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:08:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:08:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:08:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10646
2021/12/04 16:08:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebKVOpsOnUpdateBucketOpNonDefaultSettings (180.19s)
=== RUN   TestEventingSwapRebOnUpdateBucketOp
2021/12/04 16:08:56 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 16:08:56 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 16:08:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:08:56 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebOnUpdateBucketOp not found"
 }
}
2021/12/04 16:08:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:09:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:09:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:09:10 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:09:10 eventingSetup.go:125: Deploying app: TestEventingSwapRebOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:09:11 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 16:09:16 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:09:21 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 16:09:26 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 16:09:31 clusterSetup.go:346: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get deployed
2021/12/04 16:09:31 clusterSetup.go:354: App: TestEventingSwapRebOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:09:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:09:32 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:09:37 Rebalance progress: 12.5
2021/12/04 16:09:42 Rebalance progress: 12.5
2021/12/04 16:09:47 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:09:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:09:47 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:09:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:09:52 Rebalance progress: 12.5
2021/12/04 16:09:57 Rebalance progress: 12.5
2021/12/04 16:10:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:10:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:10:07 Rebalance progress: 12.5
2021/12/04 16:10:12 Rebalance progress: 12.5
2021/12/04 16:10:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:10:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:10:22 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 16:10:27 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 16:10:32 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 16:10:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebOnUpdateBucketOp to get un-deployed
2021/12/04 16:10:37 clusterSetup.go:405: App: TestEventingSwapRebOnUpdateBucketOp got un-deployed
2021/12/04 16:10:37 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:10:37 eventingSetup.go:418: Function: TestEventingSwapRebOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebOnUpdateBucketOp deleting in the background"
}
2021/12/04 16:10:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:10:49 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6098
2021/12/04 16:10:49 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:10:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:10:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:11:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5692
2021/12/04 16:11:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 5692
2021/12/04 16:11:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebOnUpdateBucketOp (139.65s)
=== RUN   TestMetaRollbackWithEventingReb
2021/12/04 16:11:11 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:11:11 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:11:16 Rebalance progress: 100
2021/12/04 16:11:21 Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 16:11:21 App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 16:11:21 No eventing-con process running
2021/12/04 16:11:21 Function: TestMetaRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaRollbackWithEventingReb not found"
 }
}
2021/12/04 16:11:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:11:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:11:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:11:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:11:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:11:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:11:35 Deploying app: TestMetaRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:11:36 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 16:11:41 Starting bucket ops to source bucket
2021/12/04 16:11:46 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 16:11:51 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 16:11:56 Waiting for app: TestMetaRollbackWithEventingReb to get deployed
2021/12/04 16:11:56 App: TestMetaRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:11:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:11:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:11:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:11:56 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:12:01 Rebalance progress: 12.5
2021/12/04 16:12:06 Rebalance progress: 12.5
2021/12/04 16:12:11 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:12:11 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:12:11 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 16:12:11 bucketOps.go:134: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 16:12:16 Rebalance progress: 12.5
2021/12/04 16:12:21 Rebalance progress: 12.5
2021/12/04 16:12:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:12:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:12:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:12:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:12:32 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 16:12:37 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 16:12:42 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 16:12:47 clusterSetup.go:397: Waiting for app: TestMetaRollbackWithEventingReb to get un-deployed
2021/12/04 16:12:47 clusterSetup.go:405: App: TestMetaRollbackWithEventingReb got un-deployed
2021/12/04 16:12:47 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:12:47 eventingSetup.go:418: Function: TestMetaRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaRollbackWithEventingReb deleting in the background"
}
2021/12/04 16:12:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:12:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:12:59 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4530
2021/12/04 16:12:59 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:13:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:13:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:13:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1818
2021/12/04 16:13:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1818
2021/12/04 16:13:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaRollbackWithEventingReb (129.34s)
=== RUN   TestMetaPartialRollbackWithEventingReb
2021/12/04 16:13:20 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:13:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:13:26 Rebalance progress: 100
2021/12/04 16:13:31 Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 16:13:31 App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 16:13:31 No eventing-con process running
2021/12/04 16:13:31 Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestMetaPartialRollbackWithEventingReb not found"
 }
}
2021/12/04 16:13:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:13:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:13:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:13:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:13:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:13:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:13:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:13:44 Deploying app: TestMetaPartialRollbackWithEventingReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:13:46 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestMetaPartialRollbackWithEventingReb' in metakv",
  "warnings": null
 }
}
2021/12/04 16:13:51 Starting bucket ops to source bucket
2021/12/04 16:13:56 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 16:14:01 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 16:14:06 Waiting for app: TestMetaPartialRollbackWithEventingReb to get deployed
2021/12/04 16:14:06 App: TestMetaPartialRollbackWithEventingReb got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:14:06 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:14:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:14:11 Rebalance progress: 12.5
2021/12/04 16:14:16 Rebalance progress: 12.5
2021/12/04 16:14:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:14:21 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 16:14:21 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 16:14:26 Rebalance progress: 12.5
2021/12/04 16:14:31 Rebalance progress: 12.5
2021/12/04 16:14:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:14:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:14:36 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:14:36 bucketOps.go:65: Metakv lookup failed, err: Get "/_metakv/eventing/tempApps/TestMetaPartialRollbackWithEventingReb/0": Unable to initialize cbauth's revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
2021/12/04 16:14:41 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 16:14:46 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 16:14:51 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 16:14:56 clusterSetup.go:397: Waiting for app: TestMetaPartialRollbackWithEventingReb to get un-deployed
2021/12/04 16:14:56 clusterSetup.go:405: App: TestMetaPartialRollbackWithEventingReb got un-deployed
2021/12/04 16:14:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:14:56 eventingSetup.go:418: Function: TestMetaPartialRollbackWithEventingReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestMetaPartialRollbackWithEventingReb deleting in the background"
}
2021/12/04 16:14:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 36569
2021/12/04 16:15:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:15:10 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:15 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestMetaPartialRollbackWithEventingReb (119.43s)
=== RUN   TestEventingRebMultipleHandlersOneByOne
2021/12/04 16:15:25 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 16:15:25 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 16:15:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:15:25 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op not found"
 }
}
2021/12/04 16:15:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:15:33 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:43 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:15:45 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:15:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:15:45 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:15:46 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersOneByOnebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 16:15:51 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:15:56 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 16:16:01 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 16:16:06 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get deployed
2021/12/04 16:16:06 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got deployed
2021/12/04 16:16:11 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 16:16:16 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 16:16:21 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get deployed
2021/12/04 16:16:21 clusterSetup.go:354: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:16:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:16:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:16:26 Rebalance progress: 12.5
2021/12/04 16:16:31 Rebalance progress: 12.5
2021/12/04 16:16:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:16:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:16:37 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:16:42 Rebalance progress: 12.5
2021/12/04 16:16:47 Rebalance progress: 12.5
2021/12/04 16:16:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:16:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:16:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 16:16:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:16:57 Rebalance progress: 12.5
2021/12/04 16:17:02 Rebalance progress: 12.5
2021/12/04 16:17:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:17:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:17:12 Rebalance progress: 12.5
2021/12/04 16:17:17 Rebalance progress: 12.5
2021/12/04 16:17:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:17:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:17:27 Rebalance progress: 12.5
2021/12/04 16:17:32 Rebalance progress: 12.5
2021/12/04 16:17:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:17:37 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 16:17:42 Rebalance progress: 12.5
2021/12/04 16:17:47 Rebalance progress: 12.5
2021/12/04 16:17:52 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:17:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:17:53 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:17:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 16:18:03 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 16:18:08 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 16:18:13 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op to get un-deployed
2021/12/04 16:18:13 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op got un-deployed
2021/12/04 16:18:13 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:18:13 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnesys_test_bucket_op deleting in the background"
}
2021/12/04 16:18:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:18:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:18:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12156
2021/12/04 16:18:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:18:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:18:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:18:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12156
2021/12/04 16:18:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:18:38 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:18:43 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 16:18:48 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 16:18:53 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 16:18:58 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update to get un-deployed
2021/12/04 16:18:58 clusterSetup.go:405: App: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update got un-deployed
2021/12/04 16:18:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:18:58 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersOneByOnebucket_op_on_update deleting in the background"
}
2021/12/04 16:19:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:19:07 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:12 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersOneByOne (237.19s)
=== RUN   TestEventingRebMultipleHandlersAllAtOnce
2021/12/04 16:19:22 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 16:19:22 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 16:19:22 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:19:22 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op not found"
 }
}
2021/12/04 16:19:24 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:29 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:19:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:41 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:19:42 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:19:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:19:42 eventingSetup.go:125: Deploying app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:19:43 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 16:19:48 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:19:53 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 16:19:58 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 16:20:03 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get deployed
2021/12/04 16:20:03 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got deployed
2021/12/04 16:20:08 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 16:20:13 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 16:20:18 clusterSetup.go:346: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get deployed
2021/12/04 16:20:18 clusterSetup.go:354: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:20:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:20:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:20:19 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: eventing to the cluster
2021/12/04 16:20:19 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:20:24 Rebalance progress: 12.5
2021/12/04 16:20:29 Rebalance progress: 12.5
2021/12/04 16:20:34 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:20:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 16:20:39 Rebalance progress: 12.5
2021/12/04 16:20:44 Rebalance progress: 12.5
2021/12/04 16:20:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:20:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:20:49 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:20:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 16:20:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 16:21:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 16:21:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op to get un-deployed
2021/12/04 16:21:09 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op got un-deployed
2021/12/04 16:21:09 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:21:09 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncesys_test_bucket_op deleting in the background"
}
2021/12/04 16:21:13 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:21:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:21:23 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6107
2021/12/04 16:21:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6107
2021/12/04 16:21:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:21:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:21:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:21:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:21:35 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:21:40 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 16:21:45 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 16:21:50 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 16:21:55 clusterSetup.go:397: Waiting for app: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update to get un-deployed
2021/12/04 16:21:55 clusterSetup.go:405: App: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update got un-deployed
2021/12/04 16:21:55 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:21:55 eventingSetup.go:418: Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultipleHandlersAllAtOncebucket_op_on_update deleting in the background"
}
2021/12/04 16:21:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:22:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultipleHandlersAllAtOnce (176.60s)
=== RUN   TestEventingSwapRebMultipleHandlers
2021/12/04 16:22:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 16:22:19 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 16:22:19 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:22:19 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update not found"
 }
}
2021/12/04 16:22:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:22:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:22:37 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 16:22:37 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 16:22:37 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:22:37 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op not found"
 }
}
2021/12/04 16:22:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:22:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:51 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:51 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlersbucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:22:52 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlersbucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:22:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:22:52 eventingSetup.go:125: Deploying app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:22:53 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebMultipleHandlerssys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 16:22:58 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:23:03 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 16:23:08 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 16:23:13 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get deployed
2021/12/04 16:23:13 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got deployed
2021/12/04 16:23:18 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 16:23:23 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 16:23:28 clusterSetup.go:346: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get deployed
2021/12/04 16:23:28 clusterSetup.go:354: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:23:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:23:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:23:34 Rebalance progress: 12.5
2021/12/04 16:23:39 Rebalance progress: 12.5
2021/12/04 16:23:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:23:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:23:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:23:49 Rebalance progress: 12.5
2021/12/04 16:23:54 Rebalance progress: 12.5
2021/12/04 16:23:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:23:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:23:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:24:04 Rebalance progress: 12.5
2021/12/04 16:24:09 Rebalance progress: 12.5
2021/12/04 16:24:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:24:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:24:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:24:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:24:19 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 16:24:24 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 16:24:29 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 16:24:34 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlersbucket_op_on_update to get un-deployed
2021/12/04 16:24:34 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlersbucket_op_on_update got un-deployed
2021/12/04 16:24:34 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:24:35 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlersbucket_op_on_update deleting in the background"
}
2021/12/04 16:24:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:24:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:24:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7604
2021/12/04 16:24:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:24:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:24:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:25:00 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1656
2021/12/04 16:25:05 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1656
2021/12/04 16:25:05 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:25:05 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:25:10 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 16:25:15 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 16:25:20 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 16:25:25 clusterSetup.go:397: Waiting for app: TestEventingSwapRebMultipleHandlerssys_test_bucket_op to get un-deployed
2021/12/04 16:25:25 clusterSetup.go:405: App: TestEventingSwapRebMultipleHandlerssys_test_bucket_op got un-deployed
2021/12/04 16:25:25 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:25:25 eventingSetup.go:418: Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebMultipleHandlerssys_test_bucket_op deleting in the background"
}
2021/12/04 16:25:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:25:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:25:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:25:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:25:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:25:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebMultipleHandlers (210.45s)
=== RUN   TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 16:25:49 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:25:49 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:25:49 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:25:49 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 16:25:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:25:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:25:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:25:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:26:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:26:08 eventingSetup.go:125: Deploying app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:26:09 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 16:26:14 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:26:19 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:26:24 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:26:29 clusterSetup.go:346: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:26:29 clusterSetup.go:354: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:26:29 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:26:29 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:26:49 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:26:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:20 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:27:20 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:27:25 Rebalance progress: 12.5
2021/12/04 16:27:30 Rebalance progress: 12.5
2021/12/04 16:27:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:27:35 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:27:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:27:41 Rebalance progress: 12.5
2021/12/04 16:27:46 Rebalance progress: 12.5
2021/12/04 16:27:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:27:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:27:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:27:56 Rebalance progress: 12.5
2021/12/04 16:28:01 Rebalance progress: 12.5
2021/12/04 16:28:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:28:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:28:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:28:06 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:28:11 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:28:16 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:28:21 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:28:26 clusterSetup.go:397: Waiting for app: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:28:26 clusterSetup.go:405: App: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:28:26 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:28:26 eventingSetup.go:418: Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 16:28:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:28:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:28:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 11178
2021/12/04 16:28:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:28:39 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:28:44 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:28:44 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.46s)
=== RUN   TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 16:28:54 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:28:54 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:28:55 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:28:55 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne not found"
 }
}
2021/12/04 16:28:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:29:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:29:03 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:08 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:29:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:29:13 eventingSetup.go:125: Deploying app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:29:14 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne' in metakv",
  "warnings": null
 }
}
2021/12/04 16:29:19 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:29:24 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:29:29 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:29:34 clusterSetup.go:346: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get deployed
2021/12/04 16:29:34 clusterSetup.go:354: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:34 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:29:34 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:29:35 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:29:55 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:29:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:30:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:30:05 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:30:25 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:26 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:30:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:30:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:30:56 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:30:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:31:07 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:31:07 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:31:27 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:38 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:31:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:31:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:31:58 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:31:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:32:09 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:32:14 Rebalance progress: 12.5
2021/12/04 16:32:19 Rebalance progress: 12.5
2021/12/04 16:32:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:32:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:32:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:32:29 Rebalance progress: 12.5
2021/12/04 16:32:34 Rebalance progress: 12.5
2021/12/04 16:32:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:32:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:32:44 Rebalance progress: 12.5
2021/12/04 16:32:49 Rebalance progress: 12.5
2021/12/04 16:32:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:32:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:32:54 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:32:59 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:33:04 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:33:09 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:33:14 clusterSetup.go:397: Waiting for app: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne to get un-deployed
2021/12/04 16:33:14 clusterSetup.go:405: App: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne got un-deployed
2021/12/04 16:33:14 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:33:14 eventingSetup.go:418: Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne deleting in the background"
}
2021/12/04 16:33:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:33:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:33:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21488
2021/12/04 16:33:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:33:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:33:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:33:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 21488
2021/12/04 16:33:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebMultiStopStartKVOpsOnUpdateBucketOpOneByOne (293.52s)
=== RUN   TestEventingFailoverOnUpdateBucketOp
2021/12/04 16:33:48 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 16:33:48 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 16:33:48 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:33:48 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingFailoverOnUpdateBucketOp not found"
 }
}
2021/12/04 16:33:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:33:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:33:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:33:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:34:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:07 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:34:07 eventingSetup.go:125: Deploying app: TestEventingFailoverOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:34:08 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingFailoverOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 16:34:13 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:34:18 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 16:34:23 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 16:34:28 clusterSetup.go:346: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get deployed
2021/12/04 16:34:28 clusterSetup.go:354: App: TestEventingFailoverOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:34:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:34:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:34:33 Rebalance progress: 12.5
2021/12/04 16:34:38 Rebalance progress: 12.5
2021/12/04 16:34:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:34:43 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:34:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:34:54 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:34:59 Rebalance progress: 12.5
2021/12/04 16:35:04 Rebalance progress: 12.5
2021/12/04 16:35:09 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:35:09 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:35:09 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:35:14 Rebalance progress: 12.5
2021/12/04 16:35:19 Rebalance progress: 12.5
2021/12/04 16:35:24 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:35:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:35:29 Rebalance progress: 12.5
2021/12/04 16:35:34 Rebalance progress: 12.5
2021/12/04 16:35:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:35:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:35:39 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:35:44 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 16:35:49 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 16:35:54 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 16:35:59 clusterSetup.go:397: Waiting for app: TestEventingFailoverOnUpdateBucketOp to get un-deployed
2021/12/04 16:35:59 clusterSetup.go:405: App: TestEventingFailoverOnUpdateBucketOp got un-deployed
2021/12/04 16:35:59 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:35:59 eventingSetup.go:418: Function: TestEventingFailoverOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingFailoverOnUpdateBucketOp deleting in the background"
}
2021/12/04 16:36:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:36:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:36:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8631
2021/12/04 16:36:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8631
2021/12/04 16:36:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:36:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:36:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:36:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8631
2021/12/04 16:36:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingFailoverOnUpdateBucketOp (169.97s)
=== RUN   TestEventingKVRebalanceOnUpdateBucketOp
2021/12/04 16:36:38 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 16:36:38 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 16:36:38 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:36:38 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp not found"
 }
}
2021/12/04 16:36:40 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:36:45 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:36:45 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:36:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:36:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:36:52 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:36:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:36:57 eventingSetup.go:125: Deploying app: TestEventingKVRebalanceOnUpdateBucketOp. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:36:58 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingKVRebalanceOnUpdateBucketOp' in metakv",
  "warnings": null
 }
}
2021/12/04 16:37:03 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:37:08 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 16:37:13 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 16:37:18 clusterSetup.go:346: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get deployed
2021/12/04 16:37:18 clusterSetup.go:354: App: TestEventingKVRebalanceOnUpdateBucketOp got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:18 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:37:18 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing,kv to the cluster
2021/12/04 16:37:18 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:37:23 Rebalance progress: 12.5
2021/12/04 16:37:28 Rebalance progress: 12.5
2021/12/04 16:37:33 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:37:33 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing,kv to the cluster
2021/12/04 16:37:34 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:37:39 Rebalance progress: 12.5
2021/12/04 16:37:44 Rebalance progress: 12.5
2021/12/04 16:37:49 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:37:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:37:49 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:37:54 Rebalance progress: 12.5
2021/12/04 16:37:59 Rebalance progress: 12.5
2021/12/04 16:38:04 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:38:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:38:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:38:04 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:38:09 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 16:38:14 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 16:38:19 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 16:38:24 clusterSetup.go:397: Waiting for app: TestEventingKVRebalanceOnUpdateBucketOp to get un-deployed
2021/12/04 16:38:24 clusterSetup.go:405: App: TestEventingKVRebalanceOnUpdateBucketOp got un-deployed
2021/12/04 16:38:24 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:38:24 eventingSetup.go:418: Function: TestEventingKVRebalanceOnUpdateBucketOp delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingKVRebalanceOnUpdateBucketOp deleting in the background"
}
2021/12/04 16:38:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:38:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:38:36 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6086
2021/12/04 16:38:36 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:38:38 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:38:43 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:38:48 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6086
2021/12/04 16:38:48 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingKVRebalanceOnUpdateBucketOp (139.60s)
=== RUN   TestEventingRebInWithOps
2021/12/04 16:38:58 clusterSetup.go:397: Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 16:38:58 clusterSetup.go:405: App: TestEventingRebInWithOps got un-deployed
2021/12/04 16:38:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 16:38:58 eventingSetup.go:418: Function: TestEventingRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithOps not found"
 }
}
2021/12/04 16:38:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:39:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:39:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:39:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:39:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:39:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:39:16 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:39:16 eventingSetup.go:125: Deploying app: TestEventingRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:39:17 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 16:39:22 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 16:39:27 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 16:39:32 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 16:39:37 clusterSetup.go:346: Waiting for app: TestEventingRebInWithOps to get deployed
2021/12/04 16:39:37 clusterSetup.go:354: App: TestEventingRebInWithOps got deployed
2021/12/04 16:39:37 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:39:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 16:39:38 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 16:39:43 Rebalance progress: 12.5
2021/12/04 16:39:48 Rebalance progress: 12.5
2021/12/04 16:39:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:39:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:39:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:39:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:39:58 Rebalance progress: 12.5
2021/12/04 16:40:03 Rebalance progress: 12.5
2021/12/04 16:40:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:40:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:40:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:40:08 Waiting for src bucket item count to get to: 200000 curr count: 32834
2021/12/04 16:40:13 Waiting for src bucket item count to get to: 200000 curr count: 48379
2021/12/04 16:40:18 Waiting for src bucket item count to get to: 200000 curr count: 48379
2021/12/04 16:40:23 Waiting for src bucket item count to get to: 200000 curr count: 64265
2021/12/04 16:40:28 Waiting for src bucket item count to get to: 200000 curr count: 64265
2021/12/04 16:40:33 Waiting for src bucket item count to get to: 200000 curr count: 79029
2021/12/04 16:40:38 Waiting for src bucket item count to get to: 200000 curr count: 79029
2021/12/04 16:40:43 Waiting for src bucket item count to get to: 200000 curr count: 94837
2021/12/04 16:40:48 Waiting for src bucket item count to get to: 200000 curr count: 94837
2021/12/04 16:40:53 Waiting for src bucket item count to get to: 200000 curr count: 94837
2021/12/04 16:40:58 Waiting for src bucket item count to get to: 200000 curr count: 110731
2021/12/04 16:41:03 Waiting for src bucket item count to get to: 200000 curr count: 110731
2021/12/04 16:41:08 Waiting for src bucket item count to get to: 200000 curr count: 126822
2021/12/04 16:41:13 Waiting for src bucket item count to get to: 200000 curr count: 126822
2021/12/04 16:41:18 Waiting for src bucket item count to get to: 200000 curr count: 142849
2021/12/04 16:41:23 Waiting for src bucket item count to get to: 200000 curr count: 142849
2021/12/04 16:41:28 Waiting for src bucket item count to get to: 200000 curr count: 158877
2021/12/04 16:41:33 Waiting for src bucket item count to get to: 200000 curr count: 158877
2021/12/04 16:41:38 Waiting for src bucket item count to get to: 200000 curr count: 173839
2021/12/04 16:41:43 Waiting for src bucket item count to get to: 200000 curr count: 173839
2021/12/04 16:41:48 Waiting for src bucket item count to get to: 200000 curr count: 189207
2021/12/04 16:41:53 Waiting for src bucket item count to get to: 200000 curr count: 189207
2021/12/04 16:41:58 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 16:41:58 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 16:41:58 Starting bucket ops to source bucket
2021/12/04 16:45:50 Waiting for dst bucket item count to get to: 0 curr count: 10362
2021/12/04 16:45:55 Waiting for dst bucket item count to get to: 0 curr count: 10362
2021/12/04 16:45:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:45:55 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 16:45:55 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:46:00 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 16:46:05 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 16:46:10 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 16:46:15 Waiting for app: TestEventingRebInWithOps to get un-deployed
2021/12/04 16:46:15 App: TestEventingRebInWithOps got un-deployed
2021/12/04 16:46:15 No eventing-con process running
2021/12/04 16:46:15 Function: TestEventingRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithOps deleting in the background"
}
2021/12/04 16:46:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:46:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:46:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:46:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:46:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:46:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithOps (461.29s)
=== RUN   TestEventingRebOutWithOps
2021/12/04 16:46:34 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:46:34 Removing node(s): [] from the cluster
2021/12/04 16:46:39 Rebalance progress: 100
2021/12/04 16:46:44 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 16:46:44 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 16:46:44 No eventing-con process running
2021/12/04 16:46:44 Function: TestEventingRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithOps not found"
 }
}
2021/12/04 16:46:46 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:46:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:46:51 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:46:53 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:46:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:46:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:47:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:47:03 Deploying app: TestEventingRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:47:04 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 16:47:09 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 16:47:14 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 16:47:19 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 16:47:24 Waiting for app: TestEventingRebOutWithOps to get deployed
2021/12/04 16:47:24 App: TestEventingRebOutWithOps got deployed
2021/12/04 16:47:24 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:47:24 Starting bucket ops to source bucket
2021/12/04 16:47:29 Rebalance progress: 12.5
2021/12/04 16:47:34 Rebalance progress: 12.5
2021/12/04 16:47:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:47:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:47:39 Waiting for src bucket item count to get to: 200000 curr count: 7409
2021/12/04 16:47:44 Waiting for src bucket item count to get to: 200000 curr count: 22951
2021/12/04 16:47:49 Waiting for src bucket item count to get to: 200000 curr count: 22951
2021/12/04 16:47:54 Waiting for src bucket item count to get to: 200000 curr count: 38843
2021/12/04 16:47:59 Waiting for src bucket item count to get to: 200000 curr count: 38843
2021/12/04 16:48:04 Waiting for src bucket item count to get to: 200000 curr count: 54961
2021/12/04 16:48:09 Waiting for src bucket item count to get to: 200000 curr count: 54961
2021/12/04 16:48:14 Waiting for src bucket item count to get to: 200000 curr count: 54961
2021/12/04 16:48:19 Waiting for src bucket item count to get to: 200000 curr count: 71285
2021/12/04 16:48:24 Waiting for src bucket item count to get to: 200000 curr count: 71285
2021/12/04 16:48:29 Waiting for src bucket item count to get to: 200000 curr count: 86347
2021/12/04 16:48:34 Waiting for src bucket item count to get to: 200000 curr count: 86347
2021/12/04 16:48:39 Waiting for src bucket item count to get to: 200000 curr count: 102295
2021/12/04 16:48:44 Waiting for src bucket item count to get to: 200000 curr count: 118434
2021/12/04 16:48:49 Waiting for src bucket item count to get to: 200000 curr count: 118434
2021/12/04 16:48:54 Waiting for src bucket item count to get to: 200000 curr count: 134404
2021/12/04 16:48:59 Waiting for src bucket item count to get to: 200000 curr count: 134404
2021/12/04 16:49:04 Waiting for src bucket item count to get to: 200000 curr count: 150233
2021/12/04 16:49:09 Waiting for src bucket item count to get to: 200000 curr count: 150233
2021/12/04 16:49:14 Waiting for src bucket item count to get to: 200000 curr count: 166083
2021/12/04 16:49:19 Waiting for src bucket item count to get to: 200000 curr count: 166083
2021/12/04 16:49:24 Waiting for src bucket item count to get to: 200000 curr count: 180865
2021/12/04 16:49:29 Waiting for src bucket item count to get to: 200000 curr count: 180865
2021/12/04 16:49:34 Waiting for src bucket item count to get to: 200000 curr count: 196377
2021/12/04 16:49:39 Waiting for src bucket item count to get to: 200000 curr count: 196377
2021/12/04 16:49:44 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 16:49:44 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 16:49:44 Starting bucket ops to source bucket
2021/12/04 16:53:36 Waiting for dst bucket item count to get to: 0 curr count: 20432
2021/12/04 16:53:41 Waiting for dst bucket item count to get to: 0 curr count: 2506
2021/12/04 16:53:46 Waiting for dst bucket item count to get to: 0 curr count: 2506
2021/12/04 16:53:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:53:46 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 16:53:46 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 16:53:51 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 16:53:56 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 16:54:01 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 16:54:06 Waiting for app: TestEventingRebOutWithOps to get un-deployed
2021/12/04 16:54:06 App: TestEventingRebOutWithOps got un-deployed
2021/12/04 16:54:06 No eventing-con process running
2021/12/04 16:54:06 Function: TestEventingRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithOps deleting in the background"
}
2021/12/04 16:54:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:54:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:54:13 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:54:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:54:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:54:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithOps (470.63s)
=== RUN   TestEventingSwapRebWithOps
2021/12/04 16:54:29 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 16:54:30 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 16:54:30 No eventing-con process running
2021/12/04 16:54:30 Function: TestEventingSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithOps not found"
 }
}
2021/12/04 16:54:31 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:54:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:54:36 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 16:54:38 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:54:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:54:43 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:54:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 16:54:48 Deploying app: TestEventingSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 16:54:49 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 16:54:54 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 16:54:59 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 16:55:04 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 16:55:09 Waiting for app: TestEventingSwapRebWithOps to get deployed
2021/12/04 16:55:09 App: TestEventingSwapRebWithOps got deployed
2021/12/04 16:55:09 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 16:55:09 Starting bucket ops to source bucket
2021/12/04 16:55:10 Removing node(s): [] from the cluster
2021/12/04 16:55:15 Rebalance progress: 12.5
2021/12/04 16:55:20 Rebalance progress: 12.5
2021/12/04 16:55:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:55:25 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 16:55:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 16:55:30 Rebalance progress: 12.5
2021/12/04 16:55:35 Rebalance progress: 12.5
2021/12/04 16:55:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:55:40 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 16:55:45 Rebalance progress: 12.5
2021/12/04 16:55:50 Rebalance progress: 12.5
2021/12/04 16:55:55 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 16:55:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 16:55:56 Waiting for src bucket item count to get to: 200000 curr count: 60738
2021/12/04 16:56:01 Waiting for src bucket item count to get to: 200000 curr count: 60738
2021/12/04 16:56:06 Waiting for src bucket item count to get to: 200000 curr count: 75706
2021/12/04 16:56:11 Waiting for src bucket item count to get to: 200000 curr count: 91384
2021/12/04 16:56:16 Waiting for src bucket item count to get to: 200000 curr count: 91384
2021/12/04 16:56:21 Waiting for src bucket item count to get to: 200000 curr count: 107387
2021/12/04 16:56:26 Waiting for src bucket item count to get to: 200000 curr count: 107387
2021/12/04 16:56:31 Waiting for src bucket item count to get to: 200000 curr count: 123058
2021/12/04 16:56:36 Waiting for src bucket item count to get to: 200000 curr count: 123058
2021/12/04 16:56:41 Waiting for src bucket item count to get to: 200000 curr count: 138831
2021/12/04 16:56:46 Waiting for src bucket item count to get to: 200000 curr count: 138831
2021/12/04 16:56:51 Waiting for src bucket item count to get to: 200000 curr count: 154771
2021/12/04 16:56:56 Waiting for src bucket item count to get to: 200000 curr count: 154771
2021/12/04 16:57:01 Waiting for src bucket item count to get to: 200000 curr count: 169473
2021/12/04 16:57:06 Waiting for src bucket item count to get to: 200000 curr count: 169473
2021/12/04 16:57:11 Waiting for src bucket item count to get to: 200000 curr count: 184909
2021/12/04 16:57:16 Waiting for src bucket item count to get to: 200000 curr count: 184909
2021/12/04 16:57:21 Waiting for src bucket item count to get to: 200000 curr count: 184909
2021/12/04 16:57:26 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 16:57:26 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 16:57:26 Starting bucket ops to source bucket
2021/12/04 17:01:18 Waiting for dst bucket item count to get to: 0 curr count: 24459
2021/12/04 17:01:23 Waiting for dst bucket item count to get to: 0 curr count: 6478
2021/12/04 17:01:28 Waiting for dst bucket item count to get to: 0 curr count: 6478
2021/12/04 17:01:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:01:28 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:01:28 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:01:33 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 17:01:38 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 17:01:43 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 17:01:48 Waiting for app: TestEventingSwapRebWithOps to get un-deployed
2021/12/04 17:01:48 App: TestEventingSwapRebWithOps got un-deployed
2021/12/04 17:01:48 No eventing-con process running
2021/12/04 17:01:48 Function: TestEventingSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithOps deleting in the background"
}
2021/12/04 17:01:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:01:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:01:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:01:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:02:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:02:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithOps (462.26s)
=== RUN   TestKVRebInWithOps
2021/12/04 17:02:12 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 17:02:12 App: TestKVRebInWithOps got un-deployed
2021/12/04 17:02:12 No eventing-con process running
2021/12/04 17:02:12 Function: TestKVRebInWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithOps not found"
 }
}
2021/12/04 17:02:14 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:02:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:02:19 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:02:20 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:02:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:02:25 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:02:30 Deploying app: TestKVRebInWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:02:31 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 17:02:36 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 17:02:42 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 17:02:47 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 17:02:52 Waiting for app: TestKVRebInWithOps to get deployed
2021/12/04 17:02:52 App: TestKVRebInWithOps got deployed
2021/12/04 17:02:52 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:02:52 Starting bucket ops to source bucket
2021/12/04 17:02:52 Removing node(s): [] from the cluster
2021/12/04 17:02:57 Rebalance progress: 12.5
2021/12/04 17:03:02 Rebalance progress: 12.5
2021/12/04 17:03:07 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:03:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:03:07 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:03:12 Rebalance progress: 12.5
2021/12/04 17:03:17 Rebalance progress: 12.5
2021/12/04 17:03:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:03:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:03:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:03:22 Waiting for src bucket item count to get to: 200000 curr count: 41851
2021/12/04 17:03:27 Waiting for src bucket item count to get to: 200000 curr count: 41851
2021/12/04 17:03:32 Waiting for src bucket item count to get to: 200000 curr count: 57557
2021/12/04 17:03:37 Waiting for src bucket item count to get to: 200000 curr count: 57557
2021/12/04 17:03:42 Waiting for src bucket item count to get to: 200000 curr count: 72379
2021/12/04 17:03:47 Waiting for src bucket item count to get to: 200000 curr count: 72379
2021/12/04 17:03:52 Waiting for src bucket item count to get to: 200000 curr count: 72379
2021/12/04 17:03:57 Waiting for src bucket item count to get to: 200000 curr count: 88060
2021/12/04 17:04:02 Waiting for src bucket item count to get to: 200000 curr count: 88060
2021/12/04 17:04:07 Waiting for src bucket item count to get to: 200000 curr count: 104281
2021/12/04 17:04:12 Waiting for src bucket item count to get to: 200000 curr count: 104281
2021/12/04 17:04:17 Waiting for src bucket item count to get to: 200000 curr count: 120039
2021/12/04 17:04:22 Waiting for src bucket item count to get to: 200000 curr count: 120039
2021/12/04 17:04:27 Waiting for src bucket item count to get to: 200000 curr count: 135756
2021/12/04 17:04:32 Waiting for src bucket item count to get to: 200000 curr count: 135756
2021/12/04 17:04:37 Waiting for src bucket item count to get to: 200000 curr count: 151401
2021/12/04 17:04:42 Waiting for src bucket item count to get to: 200000 curr count: 151401
2021/12/04 17:04:47 Waiting for src bucket item count to get to: 200000 curr count: 166134
2021/12/04 17:04:52 Waiting for src bucket item count to get to: 200000 curr count: 166134
2021/12/04 17:04:57 Waiting for src bucket item count to get to: 200000 curr count: 181678
2021/12/04 17:05:02 Waiting for src bucket item count to get to: 200000 curr count: 181678
2021/12/04 17:05:07 Waiting for src bucket item count to get to: 200000 curr count: 197765
2021/12/04 17:05:12 Waiting for src bucket item count to get to: 200000 curr count: 197765
2021/12/04 17:05:17 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 17:05:17 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 17:05:17 Starting bucket ops to source bucket
2021/12/04 17:09:09 Waiting for dst bucket item count to get to: 0 curr count: 8487
2021/12/04 17:09:14 Waiting for dst bucket item count to get to: 0 curr count: 8487
2021/12/04 17:09:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:09:14 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:09:14 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:09:19 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 17:09:24 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 17:09:29 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 17:09:34 Waiting for app: TestKVRebInWithOps to get un-deployed
2021/12/04 17:09:34 App: TestKVRebInWithOps got un-deployed
2021/12/04 17:09:34 No eventing-con process running
2021/12/04 17:09:34 Function: TestKVRebInWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithOps deleting in the background"
}
2021/12/04 17:09:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:09:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:09:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:09:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:09:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:09:48 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithOps (466.06s)
=== RUN   TestKVRebOutWithOps
2021/12/04 17:09:53 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:09:53 Removing node(s): [] from the cluster
2021/12/04 17:09:58 Rebalance progress: 100
2021/12/04 17:10:03 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 17:10:03 App: TestKVRebOutWithOps got un-deployed
2021/12/04 17:10:03 No eventing-con process running
2021/12/04 17:10:03 Function: TestKVRebOutWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithOps not found"
 }
}
2021/12/04 17:10:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:10:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:10:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:10:12 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:10:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:10:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:10:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:10:22 Deploying app: TestKVRebOutWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:10:23 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 17:10:28 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 17:10:33 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 17:10:38 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 17:10:43 Waiting for app: TestKVRebOutWithOps to get deployed
2021/12/04 17:10:43 App: TestKVRebOutWithOps got deployed
2021/12/04 17:10:43 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:10:43 Starting bucket ops to source bucket
2021/12/04 17:10:48 Rebalance progress: 12.5
2021/12/04 17:10:53 Rebalance progress: 12.5
2021/12/04 17:10:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:10:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:10:58 Waiting for src bucket item count to get to: 200000 curr count: 9043
2021/12/04 17:11:03 Waiting for src bucket item count to get to: 200000 curr count: 24951
2021/12/04 17:11:08 Waiting for src bucket item count to get to: 200000 curr count: 24951
2021/12/04 17:11:13 Waiting for src bucket item count to get to: 200000 curr count: 40896
2021/12/04 17:11:18 Waiting for src bucket item count to get to: 200000 curr count: 40896
2021/12/04 17:11:23 Waiting for src bucket item count to get to: 200000 curr count: 56963
2021/12/04 17:11:28 Waiting for src bucket item count to get to: 200000 curr count: 56963
2021/12/04 17:11:33 Waiting for src bucket item count to get to: 200000 curr count: 56963
2021/12/04 17:11:38 Waiting for src bucket item count to get to: 200000 curr count: 72213
2021/12/04 17:11:43 Waiting for src bucket item count to get to: 200000 curr count: 72213
2021/12/04 17:11:48 Waiting for src bucket item count to get to: 200000 curr count: 87839
2021/12/04 17:11:53 Waiting for src bucket item count to get to: 200000 curr count: 87839
2021/12/04 17:11:58 Waiting for src bucket item count to get to: 200000 curr count: 103735
2021/12/04 17:12:03 Waiting for src bucket item count to get to: 200000 curr count: 103735
2021/12/04 17:12:08 Waiting for src bucket item count to get to: 200000 curr count: 119888
2021/12/04 17:12:13 Waiting for src bucket item count to get to: 200000 curr count: 119888
2021/12/04 17:12:18 Waiting for src bucket item count to get to: 200000 curr count: 136060
2021/12/04 17:12:23 Waiting for src bucket item count to get to: 200000 curr count: 136060
2021/12/04 17:12:28 Waiting for src bucket item count to get to: 200000 curr count: 152239
2021/12/04 17:12:33 Waiting for src bucket item count to get to: 200000 curr count: 152239
2021/12/04 17:12:38 Waiting for src bucket item count to get to: 200000 curr count: 167815
2021/12/04 17:12:43 Waiting for src bucket item count to get to: 200000 curr count: 167815
2021/12/04 17:12:48 Waiting for src bucket item count to get to: 200000 curr count: 183373
2021/12/04 17:12:53 Waiting for src bucket item count to get to: 200000 curr count: 183373
2021/12/04 17:12:58 Waiting for src bucket item count to get to: 200000 curr count: 199097
2021/12/04 17:13:03 Waiting for src bucket item count to get to: 200000 curr count: 199097
2021/12/04 17:13:08 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 17:13:08 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 17:13:08 Starting bucket ops to source bucket
2021/12/04 17:17:00 Waiting for dst bucket item count to get to: 0 curr count: 10460
2021/12/04 17:17:05 Waiting for dst bucket item count to get to: 0 curr count: 10460
2021/12/04 17:17:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:17:05 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:17:05 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:17:10 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 17:17:15 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 17:17:20 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 17:17:25 Waiting for app: TestKVRebOutWithOps to get un-deployed
2021/12/04 17:17:25 App: TestKVRebOutWithOps got un-deployed
2021/12/04 17:17:25 No eventing-con process running
2021/12/04 17:17:25 Function: TestKVRebOutWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithOps deleting in the background"
}
2021/12/04 17:17:27 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:17:32 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:17:32 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:17:34 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:17:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:17:39 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithOps (470.98s)
=== RUN   TestKVSwapRebWithOps
2021/12/04 17:17:49 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 17:17:49 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 17:17:49 No eventing-con process running
2021/12/04 17:17:49 Function: TestKVSwapRebWithOps delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithOps not found"
 }
}
2021/12/04 17:17:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:17:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:17:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:17:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:18:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:07 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:18:07 Deploying app: TestKVSwapRebWithOps. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:18:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithOps' in metakv",
  "warnings": null
 }
}
2021/12/04 17:18:14 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 17:18:19 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 17:18:24 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 17:18:29 Waiting for app: TestKVSwapRebWithOps to get deployed
2021/12/04 17:18:29 App: TestKVSwapRebWithOps got deployed
2021/12/04 17:18:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:18:29 Starting bucket ops to source bucket
2021/12/04 17:18:29 Removing node(s): [] from the cluster
2021/12/04 17:18:34 Rebalance progress: 12.5
2021/12/04 17:18:39 Rebalance progress: 12.5
2021/12/04 17:18:44 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:18:44 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 17:18:44 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:18:49 Rebalance progress: 12.5
2021/12/04 17:18:54 Rebalance progress: 12.5
2021/12/04 17:18:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:18:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:18:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 17:19:04 Rebalance progress: 12.5
2021/12/04 17:19:09 Rebalance progress: 12.5
2021/12/04 17:19:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:19:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:19:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:19:14 Waiting for src bucket item count to get to: 200000 curr count: 61599
2021/12/04 17:19:19 Waiting for src bucket item count to get to: 200000 curr count: 61599
2021/12/04 17:19:24 Waiting for src bucket item count to get to: 200000 curr count: 76554
2021/12/04 17:19:29 Waiting for src bucket item count to get to: 200000 curr count: 76554
2021/12/04 17:19:34 Waiting for src bucket item count to get to: 200000 curr count: 92421
2021/12/04 17:19:39 Waiting for src bucket item count to get to: 200000 curr count: 92421
2021/12/04 17:19:44 Waiting for src bucket item count to get to: 200000 curr count: 108381
2021/12/04 17:19:49 Waiting for src bucket item count to get to: 200000 curr count: 108381
2021/12/04 17:19:54 Waiting for src bucket item count to get to: 200000 curr count: 124543
2021/12/04 17:19:59 Waiting for src bucket item count to get to: 200000 curr count: 124543
2021/12/04 17:20:04 Waiting for src bucket item count to get to: 200000 curr count: 124543
2021/12/04 17:20:09 Waiting for src bucket item count to get to: 200000 curr count: 140569
2021/12/04 17:20:14 Waiting for src bucket item count to get to: 200000 curr count: 140569
2021/12/04 17:20:20 Waiting for src bucket item count to get to: 200000 curr count: 156634
2021/12/04 17:20:25 Waiting for src bucket item count to get to: 200000 curr count: 171395
2021/12/04 17:20:30 Waiting for src bucket item count to get to: 200000 curr count: 171395
2021/12/04 17:20:35 Waiting for src bucket item count to get to: 200000 curr count: 187247
2021/12/04 17:20:40 Waiting for src bucket item count to get to: 200000 curr count: 187247
2021/12/04 17:20:45 src & dst bucket item count matched up. src bucket count: 200000 dst bucket count: 200000
2021/12/04 17:20:45 Post SET ops. Expected item count: 200000 got 200000
2021/12/04 17:20:45 Starting bucket ops to source bucket
2021/12/04 17:24:37 Waiting for dst bucket item count to get to: 0 curr count: 22481
2021/12/04 17:24:42 Waiting for dst bucket item count to get to: 0 curr count: 4533
2021/12/04 17:24:47 Waiting for dst bucket item count to get to: 0 curr count: 4533
2021/12/04 17:24:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:24:47 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:24:47 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:24:52 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 17:24:57 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 17:25:02 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 17:25:07 Waiting for app: TestKVSwapRebWithOps to get un-deployed
2021/12/04 17:25:07 App: TestKVSwapRebWithOps got un-deployed
2021/12/04 17:25:07 No eventing-con process running
2021/12/04 17:25:07 Function: TestKVSwapRebWithOps delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithOps deleting in the background"
}
2021/12/04 17:25:09 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:25:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:25:14 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:25:16 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:25:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:25:21 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithOps (461.78s)
=== RUN   TestEventingRebInWithTimers
2021/12/04 17:25:31 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 17:25:31 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 17:25:31 No eventing-con process running
2021/12/04 17:25:31 Function: TestEventingRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebInWithTimers not found"
 }
}
2021/12/04 17:25:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:25:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:25:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:25:39 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:25:44 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:25:44 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:25:49 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:25:49 Deploying app: TestEventingRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:25:50 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:25:55 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 17:26:00 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 17:26:05 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 17:26:10 Waiting for app: TestEventingRebInWithTimers to get deployed
2021/12/04 17:26:10 App: TestEventingRebInWithTimers got deployed
2021/12/04 17:26:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:26:10 Starting bucket ops to source bucket
2021/12/04 17:26:10 Removing node(s): [] from the cluster
2021/12/04 17:26:16 Rebalance progress: 12.5
2021/12/04 17:26:21 Rebalance progress: 12.5
2021/12/04 17:26:26 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:26:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:26:26 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:26:31 Rebalance progress: 12.5
2021/12/04 17:26:36 Rebalance progress: 12.5
2021/12/04 17:26:41 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:26:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:26:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:26:41 Waiting for src bucket item count to get to: 50000 curr count: 34056
2021/12/04 17:26:46 Waiting for src bucket item count to get to: 50000 curr count: 34056
2021/12/04 17:26:51 Waiting for src bucket item count to get to: 50000 curr count: 45458
2021/12/04 17:26:56 Waiting for src bucket item count to get to: 50000 curr count: 45458
2021/12/04 17:27:06 Waiting for dst bucket item count to get to: 50000 curr count: 41802
2021/12/04 17:27:11 Waiting for dst bucket item count to get to: 50000 curr count: 41802
2021/12/04 17:27:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:27:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:27:11 Starting bucket ops to source bucket
2021/12/04 17:28:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:28:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:28:31 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:28:36 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:28:41 Waiting for dst bucket item count to get to: 0 curr count: 43220
2021/12/04 17:28:46 Waiting for dst bucket item count to get to: 0 curr count: 43220
2021/12/04 17:28:51 Waiting for dst bucket item count to get to: 0 curr count: 20188
2021/12/04 17:28:56 Waiting for dst bucket item count to get to: 0 curr count: 20188
2021/12/04 17:29:01 Waiting for dst bucket item count to get to: 0 curr count: 5630
2021/12/04 17:29:06 Waiting for dst bucket item count to get to: 0 curr count: 5630
2021/12/04 17:29:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:29:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:29:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:29:11 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 17:29:16 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 17:29:21 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 17:29:26 Waiting for app: TestEventingRebInWithTimers to get un-deployed
2021/12/04 17:29:26 App: TestEventingRebInWithTimers got un-deployed
2021/12/04 17:29:26 No eventing-con process running
2021/12/04 17:29:26 Function: TestEventingRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebInWithTimers deleting in the background"
}
2021/12/04 17:29:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:29:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:29:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:29:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:29:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:29:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebInWithTimers (259.07s)
=== RUN   TestEventingRebOutWithTimers
2021/12/04 17:29:45 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:29:45 Removing node(s): [] from the cluster
2021/12/04 17:29:50 Rebalance progress: 100
2021/12/04 17:29:55 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 17:29:55 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 17:29:55 No eventing-con process running
2021/12/04 17:29:55 Function: TestEventingRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingRebOutWithTimers not found"
 }
}
2021/12/04 17:29:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:30:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:30:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:30:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:30:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:30:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:30:14 Deploying app: TestEventingRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:30:15 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:30:20 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 17:30:25 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 17:30:30 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 17:30:35 Waiting for app: TestEventingRebOutWithTimers to get deployed
2021/12/04 17:30:35 App: TestEventingRebOutWithTimers got deployed
2021/12/04 17:30:35 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:30:35 Starting bucket ops to source bucket
2021/12/04 17:30:40 Rebalance progress: 12.5
2021/12/04 17:30:45 Rebalance progress: 12.5
2021/12/04 17:30:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:30:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:30:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:30:50 Waiting for src bucket item count to get to: 50000 curr count: 5561
2021/12/04 17:30:55 Waiting for src bucket item count to get to: 50000 curr count: 19405
2021/12/04 17:31:00 Waiting for src bucket item count to get to: 50000 curr count: 29353
2021/12/04 17:31:05 Waiting for src bucket item count to get to: 50000 curr count: 29353
2021/12/04 17:31:10 Waiting for src bucket item count to get to: 50000 curr count: 29353
2021/12/04 17:31:15 Waiting for src bucket item count to get to: 50000 curr count: 41267
2021/12/04 17:31:20 Waiting for src bucket item count to get to: 50000 curr count: 41267
2021/12/04 17:31:30 Waiting for dst bucket item count to get to: 50000 curr count: 41077
2021/12/04 17:31:35 Waiting for dst bucket item count to get to: 50000 curr count: 41077
2021/12/04 17:31:40 Waiting for dst bucket item count to get to: 50000 curr count: 49139
2021/12/04 17:31:45 Waiting for dst bucket item count to get to: 50000 curr count: 49139
2021/12/04 17:31:45 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:31:45 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:31:45 Starting bucket ops to source bucket
2021/12/04 17:32:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:33:00 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:33:05 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:33:10 Waiting for dst bucket item count to get to: 0 curr count: 43127
2021/12/04 17:33:15 Waiting for dst bucket item count to get to: 0 curr count: 43127
2021/12/04 17:33:20 Waiting for dst bucket item count to get to: 0 curr count: 26782
2021/12/04 17:33:25 Waiting for dst bucket item count to get to: 0 curr count: 26782
2021/12/04 17:33:30 Waiting for dst bucket item count to get to: 0 curr count: 8306
2021/12/04 17:33:35 Waiting for dst bucket item count to get to: 0 curr count: 8306
2021/12/04 17:33:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:33:35 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:33:36 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:33:41 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 17:33:46 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 17:33:51 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 17:33:56 Waiting for app: TestEventingRebOutWithTimers to get un-deployed
2021/12/04 17:33:56 App: TestEventingRebOutWithTimers got un-deployed
2021/12/04 17:33:56 No eventing-con process running
2021/12/04 17:33:56 Function: TestEventingRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingRebOutWithTimers deleting in the background"
}
2021/12/04 17:33:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:34:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:34:02 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:34:04 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:34:09 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:34:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingRebOutWithTimers (269.54s)
=== RUN   TestEventingSwapRebWithTimers
2021/12/04 17:34:19 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 17:34:19 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 17:34:19 No eventing-con process running
2021/12/04 17:34:19 Function: TestEventingSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestEventingSwapRebWithTimers not found"
 }
}
2021/12/04 17:34:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:34:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:34:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:34:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:34:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:34:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:34:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:34:38 Deploying app: TestEventingSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:34:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestEventingSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:34:44 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 17:34:49 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 17:34:54 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 17:34:59 Waiting for app: TestEventingSwapRebWithTimers to get deployed
2021/12/04 17:34:59 App: TestEventingSwapRebWithTimers got deployed
2021/12/04 17:34:59 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:34:59 Starting bucket ops to source bucket
2021/12/04 17:34:59 Removing node(s): [] from the cluster
2021/12/04 17:35:04 Rebalance progress: 12.5
2021/12/04 17:35:09 Rebalance progress: 12.5
2021/12/04 17:35:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:35:14 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 17:35:15 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:35:20 Rebalance progress: 12.5
2021/12/04 17:35:25 Rebalance progress: 12.5
2021/12/04 17:35:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:35:30 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 17:35:35 Rebalance progress: 12.5
2021/12/04 17:35:40 Rebalance progress: 12.5
2021/12/04 17:35:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:35:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:35:45 Waiting for src bucket item count to get to: 50000 curr count: 47847
2021/12/04 17:35:50 Waiting for src bucket item count to get to: 50000 curr count: 47847
2021/12/04 17:36:00 Waiting for dst bucket item count to get to: 50000 curr count: 41990
2021/12/04 17:36:05 Waiting for dst bucket item count to get to: 50000 curr count: 41990
2021/12/04 17:36:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:36:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:36:05 Starting bucket ops to source bucket
2021/12/04 17:37:15 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:37:20 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:37:25 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:37:30 Waiting for dst bucket item count to get to: 0 curr count: 45666
2021/12/04 17:37:35 Waiting for dst bucket item count to get to: 0 curr count: 45666
2021/12/04 17:37:40 Waiting for dst bucket item count to get to: 0 curr count: 32206
2021/12/04 17:37:45 Waiting for dst bucket item count to get to: 0 curr count: 32206
2021/12/04 17:37:50 Waiting for dst bucket item count to get to: 0 curr count: 21605
2021/12/04 17:37:56 Waiting for dst bucket item count to get to: 0 curr count: 21605
2021/12/04 17:37:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:37:56 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:37:56 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:38:01 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 17:38:06 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 17:38:11 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 17:38:16 Waiting for app: TestEventingSwapRebWithTimers to get un-deployed
2021/12/04 17:38:16 App: TestEventingSwapRebWithTimers got un-deployed
2021/12/04 17:38:16 No eventing-con process running
2021/12/04 17:38:16 Function: TestEventingSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestEventingSwapRebWithTimers deleting in the background"
}
2021/12/04 17:38:18 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:38:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:38:23 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:38:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:38:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:38:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestEventingSwapRebWithTimers (260.21s)
=== RUN   TestKVRebInWithTimers
2021/12/04 17:38:39 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 17:38:39 App: TestKVRebInWithTimers got un-deployed
2021/12/04 17:38:39 No eventing-con process running
2021/12/04 17:38:39 Function: TestKVRebInWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebInWithTimers not found"
 }
}
2021/12/04 17:38:41 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:38:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:38:46 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:38:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:38:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:38:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:38:58 Deploying app: TestKVRebInWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:38:59 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebInWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:39:04 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 17:39:09 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 17:39:14 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 17:39:19 Waiting for app: TestKVRebInWithTimers to get deployed
2021/12/04 17:39:19 App: TestKVRebInWithTimers got deployed
2021/12/04 17:39:19 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:39:19 Starting bucket ops to source bucket
2021/12/04 17:39:20 Removing node(s): [] from the cluster
2021/12/04 17:39:25 Rebalance progress: 12.5
2021/12/04 17:39:30 Rebalance progress: 12.5
2021/12/04 17:39:35 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:39:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:39:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:39:35 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:39:40 Rebalance progress: 12.5
2021/12/04 17:39:45 Rebalance progress: 12.5
2021/12/04 17:39:50 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:39:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:39:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:39:50 Waiting for src bucket item count to get to: 50000 curr count: 24869
2021/12/04 17:39:55 Waiting for src bucket item count to get to: 50000 curr count: 34729
2021/12/04 17:40:00 Waiting for src bucket item count to get to: 50000 curr count: 34729
2021/12/04 17:40:05 Waiting for src bucket item count to get to: 50000 curr count: 46857
2021/12/04 17:40:10 Waiting for src bucket item count to get to: 50000 curr count: 46857
2021/12/04 17:40:20 Waiting for dst bucket item count to get to: 50000 curr count: 44855
2021/12/04 17:40:25 Waiting for dst bucket item count to get to: 50000 curr count: 44855
2021/12/04 17:40:30 Waiting for dst bucket item count to get to: 50000 curr count: 49523
2021/12/04 17:40:30 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:40:30 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:40:30 Starting bucket ops to source bucket
2021/12/04 17:41:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:41:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:41:51 Waiting for dst bucket item count to get to: 0 curr count: 47563
2021/12/04 17:41:56 Waiting for dst bucket item count to get to: 0 curr count: 47563
2021/12/04 17:42:01 Waiting for dst bucket item count to get to: 0 curr count: 35145
2021/12/04 17:42:06 Waiting for dst bucket item count to get to: 0 curr count: 35145
2021/12/04 17:42:11 Waiting for dst bucket item count to get to: 0 curr count: 24321
2021/12/04 17:42:16 Waiting for dst bucket item count to get to: 0 curr count: 24321
2021/12/04 17:42:21 Waiting for dst bucket item count to get to: 0 curr count: 3033
2021/12/04 17:42:26 Waiting for dst bucket item count to get to: 0 curr count: 3033
2021/12/04 17:42:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:42:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:42:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:42:31 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 17:42:36 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 17:42:41 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 17:42:46 Waiting for app: TestKVRebInWithTimers to get un-deployed
2021/12/04 17:42:46 App: TestKVRebInWithTimers got un-deployed
2021/12/04 17:42:46 No eventing-con process running
2021/12/04 17:42:46 Function: TestKVRebInWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebInWithTimers deleting in the background"
}
2021/12/04 17:42:48 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:42:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:42:53 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:42:55 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:43:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:43:00 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebInWithTimers (270.33s)
=== RUN   TestKVRebOutWithTimers
2021/12/04 17:43:05 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:43:05 Removing node(s): [] from the cluster
2021/12/04 17:43:10 Rebalance progress: 100
2021/12/04 17:43:15 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 17:43:15 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 17:43:15 No eventing-con process running
2021/12/04 17:43:15 Function: TestKVRebOutWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVRebOutWithTimers not found"
 }
}
2021/12/04 17:43:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:43:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:43:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:43:24 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:43:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:43:29 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:43:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:43:34 Deploying app: TestKVRebOutWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:43:35 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVRebOutWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:43:40 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 17:43:45 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 17:43:50 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 17:43:55 Waiting for app: TestKVRebOutWithTimers to get deployed
2021/12/04 17:43:55 App: TestKVRebOutWithTimers got deployed
2021/12/04 17:43:55 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:43:55 Starting bucket ops to source bucket
2021/12/04 17:44:00 Rebalance progress: 12.5
2021/12/04 17:44:05 Rebalance progress: 12.5
2021/12/04 17:44:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:44:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:44:10 Waiting for src bucket item count to get to: 50000 curr count: 5726
2021/12/04 17:44:15 Waiting for src bucket item count to get to: 50000 curr count: 19807
2021/12/04 17:44:20 Waiting for src bucket item count to get to: 50000 curr count: 30295
2021/12/04 17:44:25 Waiting for src bucket item count to get to: 50000 curr count: 30295
2021/12/04 17:44:30 Waiting for src bucket item count to get to: 50000 curr count: 42611
2021/12/04 17:44:35 Waiting for src bucket item count to get to: 50000 curr count: 42611
2021/12/04 17:44:40 Waiting for src bucket item count to get to: 50000 curr count: 42611
2021/12/04 17:44:50 Waiting for dst bucket item count to get to: 50000 curr count: 39979
2021/12/04 17:44:55 Waiting for dst bucket item count to get to: 50000 curr count: 39979
2021/12/04 17:45:00 Waiting for dst bucket item count to get to: 50000 curr count: 48229
2021/12/04 17:45:05 Waiting for dst bucket item count to get to: 50000 curr count: 48229
2021/12/04 17:45:05 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:45:05 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:45:05 Starting bucket ops to source bucket
2021/12/04 17:46:16 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:46:21 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:46:26 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:46:31 Waiting for dst bucket item count to get to: 0 curr count: 48590
2021/12/04 17:46:36 Waiting for dst bucket item count to get to: 0 curr count: 48590
2021/12/04 17:46:41 Waiting for dst bucket item count to get to: 0 curr count: 34712
2021/12/04 17:46:46 Waiting for dst bucket item count to get to: 0 curr count: 34712
2021/12/04 17:46:51 Waiting for dst bucket item count to get to: 0 curr count: 12232
2021/12/04 17:46:56 Waiting for dst bucket item count to get to: 0 curr count: 12232
2021/12/04 17:47:01 Waiting for dst bucket item count to get to: 0 curr count: 155
2021/12/04 17:47:06 Waiting for dst bucket item count to get to: 0 curr count: 155
2021/12/04 17:47:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:47:06 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:47:06 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:47:11 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 17:47:16 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 17:47:21 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 17:47:26 Waiting for app: TestKVRebOutWithTimers to get un-deployed
2021/12/04 17:47:26 App: TestKVRebOutWithTimers got un-deployed
2021/12/04 17:47:26 No eventing-con process running
2021/12/04 17:47:26 Function: TestKVRebOutWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVRebOutWithTimers deleting in the background"
}
2021/12/04 17:47:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:47:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:47:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:47:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:47:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:47:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOutWithTimers (279.82s)
=== RUN   TestKVSwapRebWithTimers
2021/12/04 17:47:50 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:47:50 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 17:47:50 No eventing-con process running
2021/12/04 17:47:50 Function: TestKVSwapRebWithTimers delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestKVSwapRebWithTimers not found"
 }
}
2021/12/04 17:47:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:47:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:47:56 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:47:58 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:48:03 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:48:03 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:48:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:48:08 Deploying app: TestKVSwapRebWithTimers. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:48:09 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestKVSwapRebWithTimers' in metakv",
  "warnings": null
 }
}
2021/12/04 17:48:14 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 17:48:19 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 17:48:24 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 17:48:29 Waiting for app: TestKVSwapRebWithTimers to get deployed
2021/12/04 17:48:29 App: TestKVSwapRebWithTimers got deployed
2021/12/04 17:48:29 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 17:48:29 Starting bucket ops to source bucket
2021/12/04 17:48:30 Removing node(s): [] from the cluster
2021/12/04 17:48:35 Rebalance progress: 12.5
2021/12/04 17:48:40 Rebalance progress: 12.5
2021/12/04 17:48:45 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:48:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:48:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:48:45 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 17:48:45 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:48:50 Rebalance progress: 12.5
2021/12/04 17:48:55 Rebalance progress: 12.5
2021/12/04 17:49:00 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:49:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:49:00 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:49:00 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 17:49:05 Rebalance progress: 12.5
2021/12/04 17:49:10 Rebalance progress: 12.5
2021/12/04 17:49:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:49:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:49:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:49:16 Waiting for src bucket item count to get to: 50000 curr count: 46646
2021/12/04 17:49:26 Waiting for dst bucket item count to get to: 50000 curr count: 42775
2021/12/04 17:49:31 Waiting for dst bucket item count to get to: 50000 curr count: 42775
2021/12/04 17:49:31 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:49:31 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:49:31 Starting bucket ops to source bucket
2021/12/04 17:50:41 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:50:46 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:50:51 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:50:56 Waiting for dst bucket item count to get to: 0 curr count: 43116
2021/12/04 17:51:01 Waiting for dst bucket item count to get to: 0 curr count: 43116
2021/12/04 17:51:06 Waiting for dst bucket item count to get to: 0 curr count: 24231
2021/12/04 17:51:11 Waiting for dst bucket item count to get to: 0 curr count: 24231
2021/12/04 17:51:16 Waiting for dst bucket item count to get to: 0 curr count: 11584
2021/12/04 17:51:21 Waiting for dst bucket item count to get to: 0 curr count: 11584
2021/12/04 17:51:26 Waiting for dst bucket item count to get to: 0 curr count: 11584
2021/12/04 17:51:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:51:26 Post DELETE ops. Expected item count: 0 got 0
2021/12/04 17:51:26 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:51:31 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:37 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:42 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:47 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:52 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:57 Waiting for app: TestKVSwapRebWithTimers to get un-deployed
2021/12/04 17:51:57 App: TestKVSwapRebWithTimers got un-deployed
2021/12/04 17:51:57 No eventing-con process running
2021/12/04 17:51:57 Function: TestKVSwapRebWithTimers delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestKVSwapRebWithTimers deleting in the background"
}
2021/12/04 17:51:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:52:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:52:04 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:52:05 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:52:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:52:10 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebWithTimers (270.81s)
=== RUN   TestDuplicateTimerCreationEventingRebIn
2021/12/04 17:52:20 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:52:20 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 17:52:20 No eventing-con process running
2021/12/04 17:52:20 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebIn not found"
 }
}
2021/12/04 17:52:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:52:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:52:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:52:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:52:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:52:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:52:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:52:39 Deploying app: TestDuplicateTimerCreationEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:52:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 17:52:45 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 17:52:50 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 17:52:55 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 17:53:00 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get deployed
2021/12/04 17:53:00 App: TestDuplicateTimerCreationEventingRebIn got deployed
2021/12/04 17:53:00 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:53:00 Starting bucket ops to source bucket
2021/12/04 17:53:01 Removing node(s): [] from the cluster
2021/12/04 17:53:06 Rebalance progress: 12.5
2021/12/04 17:53:11 Rebalance progress: 12.5
2021/12/04 17:53:16 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:53:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:53:16 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:53:16 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:53:21 Rebalance progress: 12.5
2021/12/04 17:53:26 Rebalance progress: 12.5
2021/12/04 17:53:31 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:53:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:53:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:53:31 Waiting for src bucket item count to get to: 50000 curr count: 35511
2021/12/04 17:53:36 Waiting for src bucket item count to get to: 50000 curr count: 35511
2021/12/04 17:53:41 Waiting for src bucket item count to get to: 50000 curr count: 45192
2021/12/04 17:53:46 Waiting for src bucket item count to get to: 50000 curr count: 45192
2021/12/04 17:53:56 Waiting for dst bucket item count to get to: 50000 curr count: 34382
2021/12/04 17:54:01 Waiting for dst bucket item count to get to: 50000 curr count: 34382
2021/12/04 17:54:06 Waiting for dst bucket item count to get to: 50000 curr count: 45454
2021/12/04 17:54:11 Waiting for dst bucket item count to get to: 50000 curr count: 45454
2021/12/04 17:54:11 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:54:11 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:54:11 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:54:16 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:21 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:26 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:31 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:36 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:41 Waiting for app: TestDuplicateTimerCreationEventingRebIn to get un-deployed
2021/12/04 17:54:41 App: TestDuplicateTimerCreationEventingRebIn got un-deployed
2021/12/04 17:54:41 No eventing-con process running
2021/12/04 17:54:41 Function: TestDuplicateTimerCreationEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebIn deleting in the background"
}
2021/12/04 17:54:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:54:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:54:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:54:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:54:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:55:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:55:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:55:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebIn (174.45s)
=== RUN   TestDuplicateTimerCreationEventingRebOut
2021/12/04 17:55:10 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:55:11 Removing node(s): [] from the cluster
2021/12/04 17:55:16 Rebalance progress: 100
2021/12/04 17:55:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:55:21 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 17:55:21 No eventing-con process running
2021/12/04 17:55:21 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingRebOut not found"
 }
}
2021/12/04 17:55:22 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:55:27 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:55:27 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:55:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:55:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:55:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:55:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:55:39 Deploying app: TestDuplicateTimerCreationEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:55:40 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 17:55:45 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 17:55:50 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 17:55:55 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 17:56:00 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get deployed
2021/12/04 17:56:00 App: TestDuplicateTimerCreationEventingRebOut got deployed
2021/12/04 17:56:00 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:56:00 Starting bucket ops to source bucket
2021/12/04 17:56:05 Rebalance progress: 12.5
2021/12/04 17:56:10 Rebalance progress: 12.5
2021/12/04 17:56:15 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:56:15 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:56:16 Waiting for src bucket item count to get to: 50000 curr count: 11244
2021/12/04 17:56:21 Waiting for src bucket item count to get to: 50000 curr count: 24861
2021/12/04 17:56:26 Waiting for src bucket item count to get to: 50000 curr count: 24861
2021/12/04 17:56:31 Waiting for src bucket item count to get to: 50000 curr count: 35528
2021/12/04 17:56:36 Waiting for src bucket item count to get to: 50000 curr count: 35528
2021/12/04 17:56:41 Waiting for src bucket item count to get to: 50000 curr count: 45017
2021/12/04 17:56:46 Waiting for src bucket item count to get to: 50000 curr count: 45017
2021/12/04 17:56:51 Waiting for src bucket item count to get to: 50000 curr count: 45017
2021/12/04 17:57:01 Waiting for dst bucket item count to get to: 50000 curr count: 36540
2021/12/04 17:57:06 Waiting for dst bucket item count to get to: 50000 curr count: 36540
2021/12/04 17:57:11 Waiting for dst bucket item count to get to: 50000 curr count: 48108
2021/12/04 17:57:16 Waiting for dst bucket item count to get to: 50000 curr count: 48108
2021/12/04 17:57:16 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 17:57:16 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 17:57:16 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 17:57:21 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:57:26 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:57:31 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:57:36 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:57:41 Waiting for app: TestDuplicateTimerCreationEventingRebOut to get un-deployed
2021/12/04 17:57:41 App: TestDuplicateTimerCreationEventingRebOut got un-deployed
2021/12/04 17:57:41 No eventing-con process running
2021/12/04 17:57:41 Function: TestDuplicateTimerCreationEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingRebOut deleting in the background"
}
2021/12/04 17:57:43 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:57:48 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:57:53 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:57:58 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 17:57:58 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:58:00 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:58:05 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:58:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingRebOut (179.93s)
=== RUN   TestDuplicateTimerCreationEventingSwapReb
2021/12/04 17:58:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 17:58:15 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 17:58:15 No eventing-con process running
2021/12/04 17:58:15 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationEventingSwapReb not found"
 }
}
2021/12/04 17:58:17 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:58:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:58:22 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 17:58:23 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:58:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:58:28 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:58:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 17:58:33 Deploying app: TestDuplicateTimerCreationEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 17:58:34 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 17:58:39 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 17:58:45 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 17:58:50 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 17:58:55 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get deployed
2021/12/04 17:58:55 App: TestDuplicateTimerCreationEventingSwapReb got deployed
2021/12/04 17:58:55 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 17:58:55 Starting bucket ops to source bucket
2021/12/04 17:58:55 Removing node(s): [] from the cluster
2021/12/04 17:59:00 Rebalance progress: 12.5
2021/12/04 17:59:05 Rebalance progress: 12.5
2021/12/04 17:59:10 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:10 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:59:10 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 17:59:10 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 17:59:15 Rebalance progress: 12.5
2021/12/04 17:59:20 Rebalance progress: 12.5
2021/12/04 17:59:25 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:25 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:59:25 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 17:59:30 Rebalance progress: 12.5
2021/12/04 17:59:35 Rebalance progress: 12.5
2021/12/04 17:59:40 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 17:59:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 17:59:45 Waiting for dst bucket item count to get to: 50000 curr count: 27131
2021/12/04 17:59:50 Waiting for dst bucket item count to get to: 50000 curr count: 27131
2021/12/04 17:59:55 Waiting for dst bucket item count to get to: 50000 curr count: 35936
2021/12/04 18:00:00 Waiting for dst bucket item count to get to: 50000 curr count: 35936
2021/12/04 18:00:05 Waiting for dst bucket item count to get to: 50000 curr count: 48385
2021/12/04 18:00:10 Waiting for dst bucket item count to get to: 50000 curr count: 48385
2021/12/04 18:00:10 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:00:10 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:00:10 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:00:15 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:21 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:26 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:31 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:36 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:41 Waiting for app: TestDuplicateTimerCreationEventingSwapReb to get un-deployed
2021/12/04 18:00:41 App: TestDuplicateTimerCreationEventingSwapReb got un-deployed
2021/12/04 18:00:41 No eventing-con process running
2021/12/04 18:00:41 Function: TestDuplicateTimerCreationEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationEventingSwapReb deleting in the background"
}
2021/12/04 18:00:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:00:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:00:52 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:00:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:00:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:00:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:01:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:01:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:01:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationEventingSwapReb (184.54s)
=== RUN   TestDuplicateTimerCreationKVRebIn
2021/12/04 18:01:19 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:01:19 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 18:01:19 No eventing-con process running
2021/12/04 18:01:19 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebIn not found"
 }
}
2021/12/04 18:01:21 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:01:26 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:01:26 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:01:28 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:01:33 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:01:33 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:01:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:01:38 Deploying app: TestDuplicateTimerCreationKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:01:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 18:01:44 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 18:01:49 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 18:01:54 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 18:01:59 Waiting for app: TestDuplicateTimerCreationKVRebIn to get deployed
2021/12/04 18:01:59 App: TestDuplicateTimerCreationKVRebIn got deployed
2021/12/04 18:01:59 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:01:59 Starting bucket ops to source bucket
2021/12/04 18:01:59 Removing node(s): [] from the cluster
2021/12/04 18:02:04 Rebalance progress: 12.5
2021/12/04 18:02:09 Rebalance progress: 12.5
2021/12/04 18:02:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:02:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:02:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:02:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:02:20 Rebalance progress: 12.5
2021/12/04 18:02:25 Rebalance progress: 12.5
2021/12/04 18:02:30 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:02:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:02:30 Waiting for src bucket item count to get to: 50000 curr count: 25750
2021/12/04 18:02:35 Waiting for src bucket item count to get to: 50000 curr count: 36238
2021/12/04 18:02:40 Waiting for src bucket item count to get to: 50000 curr count: 36238
2021/12/04 18:02:45 Waiting for src bucket item count to get to: 50000 curr count: 46463
2021/12/04 18:02:50 Waiting for src bucket item count to get to: 50000 curr count: 46463
2021/12/04 18:03:00 Waiting for dst bucket item count to get to: 50000 curr count: 38203
2021/12/04 18:03:05 Waiting for dst bucket item count to get to: 50000 curr count: 38203
2021/12/04 18:03:10 Waiting for dst bucket item count to get to: 50000 curr count: 45315
2021/12/04 18:03:15 Waiting for dst bucket item count to get to: 50000 curr count: 45315
2021/12/04 18:03:15 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:03:15 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:03:15 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:03:20 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:25 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:30 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:35 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:40 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:45 Waiting for app: TestDuplicateTimerCreationKVRebIn to get un-deployed
2021/12/04 18:03:45 App: TestDuplicateTimerCreationKVRebIn got un-deployed
2021/12/04 18:03:45 No eventing-con process running
2021/12/04 18:03:45 Function: TestDuplicateTimerCreationKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebIn deleting in the background"
}
2021/12/04 18:03:47 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:03:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:03:57 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:03:57 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:03:59 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:04:04 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:04:09 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:04:09 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebIn (179.29s)
=== RUN   TestDuplicateTimerCreationKVRebOut
2021/12/04 18:04:14 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:04:14 Removing node(s): [] from the cluster
2021/12/04 18:04:19 Rebalance progress: 100
2021/12/04 18:04:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:04:24 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 18:04:24 No eventing-con process running
2021/12/04 18:04:24 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVRebOut not found"
 }
}
2021/12/04 18:04:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:04:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:04:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:04:33 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:04:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:04:38 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:04:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:04:43 Deploying app: TestDuplicateTimerCreationKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:04:44 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 18:04:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 18:04:54 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 18:04:59 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 18:05:04 Waiting for app: TestDuplicateTimerCreationKVRebOut to get deployed
2021/12/04 18:05:04 App: TestDuplicateTimerCreationKVRebOut got deployed
2021/12/04 18:05:04 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:05:04 Starting bucket ops to source bucket
2021/12/04 18:05:09 Rebalance progress: 12.5
2021/12/04 18:05:14 Rebalance progress: 12.5
2021/12/04 18:05:19 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:05:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:05:19 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:05:19 Waiting for src bucket item count to get to: 50000 curr count: 6721
2021/12/04 18:05:24 Waiting for src bucket item count to get to: 50000 curr count: 20532
2021/12/04 18:05:29 Waiting for src bucket item count to get to: 50000 curr count: 20532
2021/12/04 18:05:34 Waiting for src bucket item count to get to: 50000 curr count: 33623
2021/12/04 18:05:39 Waiting for src bucket item count to get to: 50000 curr count: 33623
2021/12/04 18:05:44 Waiting for src bucket item count to get to: 50000 curr count: 33623
2021/12/04 18:05:49 Waiting for src bucket item count to get to: 50000 curr count: 43857
2021/12/04 18:05:54 Waiting for src bucket item count to get to: 50000 curr count: 43857
2021/12/04 18:06:04 Waiting for dst bucket item count to get to: 50000 curr count: 29886
2021/12/04 18:06:09 Waiting for dst bucket item count to get to: 50000 curr count: 29886
2021/12/04 18:06:14 Waiting for dst bucket item count to get to: 50000 curr count: 36976
2021/12/04 18:06:19 Waiting for dst bucket item count to get to: 50000 curr count: 36976
2021/12/04 18:06:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:06:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:06:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:06:24 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:29 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:34 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:39 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:44 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:49 Waiting for app: TestDuplicateTimerCreationKVRebOut to get un-deployed
2021/12/04 18:06:49 App: TestDuplicateTimerCreationKVRebOut got un-deployed
2021/12/04 18:06:49 No eventing-con process running
2021/12/04 18:06:49 Function: TestDuplicateTimerCreationKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVRebOut deleting in the background"
}
2021/12/04 18:06:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:06:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:07:01 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:07:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:07:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:07:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:07:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:07:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:07:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVRebOut (189.39s)
=== RUN   TestDuplicateTimerCreationKVSwapReb
2021/12/04 18:07:28 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:07:28 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 18:07:28 No eventing-con process running
2021/12/04 18:07:28 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerCreationKVSwapReb not found"
 }
}
2021/12/04 18:07:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:07:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:07:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:07:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:07:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:07:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:07:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:07:47 Deploying app: TestDuplicateTimerCreationKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:07:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerCreationKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 18:07:53 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 18:07:58 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 18:08:03 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 18:08:08 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get deployed
2021/12/04 18:08:08 App: TestDuplicateTimerCreationKVSwapReb got deployed
2021/12/04 18:08:08 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:08:08 Starting bucket ops to source bucket
2021/12/04 18:08:08 Removing node(s): [] from the cluster
2021/12/04 18:08:13 Rebalance progress: 12.5
2021/12/04 18:08:18 Rebalance progress: 12.5
2021/12/04 18:08:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:08:24 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:08:24 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:08:29 Rebalance progress: 12.5
2021/12/04 18:08:34 Rebalance progress: 12.5
2021/12/04 18:08:39 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:39 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:08:39 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:08:44 Rebalance progress: 12.5
2021/12/04 18:08:49 Rebalance progress: 12.5
2021/12/04 18:08:54 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:08:54 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:08:54 Waiting for src bucket item count to get to: 50000 curr count: 48781
2021/12/04 18:08:59 Waiting for src bucket item count to get to: 50000 curr count: 48781
2021/12/04 18:09:09 Waiting for dst bucket item count to get to: 50000 curr count: 35595
2021/12/04 18:09:14 Waiting for dst bucket item count to get to: 50000 curr count: 35595
2021/12/04 18:09:19 Waiting for dst bucket item count to get to: 50000 curr count: 41858
2021/12/04 18:09:24 Waiting for dst bucket item count to get to: 50000 curr count: 41858
2021/12/04 18:09:24 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:09:24 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:09:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:09:29 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:34 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:39 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:45 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:50 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:55 Waiting for app: TestDuplicateTimerCreationKVSwapReb to get un-deployed
2021/12/04 18:09:55 App: TestDuplicateTimerCreationKVSwapReb got un-deployed
2021/12/04 18:09:55 No eventing-con process running
2021/12/04 18:09:55 Function: TestDuplicateTimerCreationKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerCreationKVSwapReb deleting in the background"
}
2021/12/04 18:09:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:10:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:10:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:10:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:10:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:10:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:10:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:10:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerCreationKVSwapReb (180.21s)
=== RUN   TestDuplicateTimerFiringEventingRebIn
2021/12/04 18:10:28 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:10:28 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 18:10:28 No eventing-con process running
2021/12/04 18:10:28 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebIn not found"
 }
}
2021/12/04 18:10:30 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:10:35 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:10:35 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:10:37 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:10:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:10:42 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:10:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:10:47 Deploying app: TestDuplicateTimerFiringEventingRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:10:48 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 18:10:53 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 18:10:58 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 18:11:03 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 18:11:08 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get deployed
2021/12/04 18:11:08 App: TestDuplicateTimerFiringEventingRebIn got deployed
2021/12/04 18:11:08 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 18:11:08 Starting bucket ops to source bucket
2021/12/04 18:11:08 Removing node(s): [] from the cluster
2021/12/04 18:11:13 Rebalance progress: 12.5
2021/12/04 18:11:18 Rebalance progress: 12.5
2021/12/04 18:11:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:11:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:11:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:11:28 Rebalance progress: 12.5
2021/12/04 18:11:33 Rebalance progress: 12.5
2021/12/04 18:11:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:11:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:11:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:11:38 Waiting for src bucket item count to get to: 50000 curr count: 27374
2021/12/04 18:11:43 Waiting for src bucket item count to get to: 50000 curr count: 36604
2021/12/04 18:11:48 Waiting for src bucket item count to get to: 50000 curr count: 36604
2021/12/04 18:11:53 Waiting for src bucket item count to get to: 50000 curr count: 49190
2021/12/04 18:11:59 Waiting for src bucket item count to get to: 50000 curr count: 49190
2021/12/04 18:12:04 Waiting for src bucket item count to get to: 50000 curr count: 49190
2021/12/04 18:12:14 Waiting for dst bucket item count to get to: 50000 curr count: 43794
2021/12/04 18:12:19 Waiting for dst bucket item count to get to: 50000 curr count: 43794
2021/12/04 18:12:19 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:12:19 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:12:19 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:12:24 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:29 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:34 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:39 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:44 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:49 Waiting for app: TestDuplicateTimerFiringEventingRebIn to get un-deployed
2021/12/04 18:12:49 App: TestDuplicateTimerFiringEventingRebIn got un-deployed
2021/12/04 18:12:49 No eventing-con process running
2021/12/04 18:12:49 Function: TestDuplicateTimerFiringEventingRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebIn deleting in the background"
}
2021/12/04 18:12:51 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:12:56 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:13:01 Waiting for dst bucket item count to get to: 0 curr count: 39872
2021/12/04 18:13:06 Waiting for dst bucket item count to get to: 0 curr count: 39872
2021/12/04 18:13:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:13:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:13:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:13:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:13:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebIn (179.47s)
=== RUN   TestDuplicateTimerFiringEventingRebOut
2021/12/04 18:13:23 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 18:13:23 Removing node(s): [] from the cluster
2021/12/04 18:13:28 Rebalance progress: 100
2021/12/04 18:13:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:13:33 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 18:13:33 No eventing-con process running
2021/12/04 18:13:33 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingRebOut not found"
 }
}
2021/12/04 18:13:35 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:13:40 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:13:40 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:13:42 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:13:47 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:13:47 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:13:52 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:13:52 Deploying app: TestDuplicateTimerFiringEventingRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:13:53 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 18:13:58 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 18:14:03 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 18:14:08 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 18:14:13 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get deployed
2021/12/04 18:14:13 App: TestDuplicateTimerFiringEventingRebOut got deployed
2021/12/04 18:14:13 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:14:13 Starting bucket ops to source bucket
2021/12/04 18:14:18 Rebalance progress: 12.5
2021/12/04 18:14:23 Rebalance progress: 12.5
2021/12/04 18:14:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:14:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:14:28 Waiting for src bucket item count to get to: 50000 curr count: 8298
2021/12/04 18:14:33 Waiting for src bucket item count to get to: 50000 curr count: 20318
2021/12/04 18:14:38 Waiting for src bucket item count to get to: 50000 curr count: 20318
2021/12/04 18:14:43 Waiting for src bucket item count to get to: 50000 curr count: 31294
2021/12/04 18:14:48 Waiting for src bucket item count to get to: 50000 curr count: 31294
2021/12/04 18:14:53 Waiting for src bucket item count to get to: 50000 curr count: 42383
2021/12/04 18:14:58 Waiting for src bucket item count to get to: 50000 curr count: 42383
2021/12/04 18:15:03 Waiting for src bucket item count to get to: 50000 curr count: 42383
2021/12/04 18:15:13 Waiting for dst bucket item count to get to: 50000 curr count: 40937
2021/12/04 18:15:18 Waiting for dst bucket item count to get to: 50000 curr count: 40937
2021/12/04 18:15:18 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:15:18 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:15:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:15:23 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:28 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:33 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:39 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:44 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:49 Waiting for app: TestDuplicateTimerFiringEventingRebOut to get un-deployed
2021/12/04 18:15:49 App: TestDuplicateTimerFiringEventingRebOut got un-deployed
2021/12/04 18:15:49 No eventing-con process running
2021/12/04 18:15:49 Function: TestDuplicateTimerFiringEventingRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingRebOut deleting in the background"
}
2021/12/04 18:15:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:15:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:16:00 Waiting for dst bucket item count to get to: 0 curr count: 24962
2021/12/04 18:16:05 Waiting for dst bucket item count to get to: 0 curr count: 24962
2021/12/04 18:16:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:16:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:16:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:16:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:16:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingRebOut (179.66s)
=== RUN   TestDuplicateTimerFiringEventingSwapReb
2021/12/04 18:16:27 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:16:27 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 18:16:27 No eventing-con process running
2021/12/04 18:16:27 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringEventingSwapReb not found"
 }
}
2021/12/04 18:16:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:16:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:16:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:16:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:16:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:16:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:16:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:16:46 Deploying app: TestDuplicateTimerFiringEventingSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:16:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringEventingSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 18:16:52 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 18:16:57 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 18:17:02 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 18:17:07 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get deployed
2021/12/04 18:17:07 App: TestDuplicateTimerFiringEventingSwapReb got deployed
2021/12/04 18:17:07 Adding node: https://127.0.0.1:19001 with role: eventing to the cluster
2021/12/04 18:17:07 Starting bucket ops to source bucket
2021/12/04 18:17:08 Removing node(s): [] from the cluster
2021/12/04 18:17:13 Rebalance progress: 12.5
2021/12/04 18:17:18 Rebalance progress: 12.5
2021/12/04 18:17:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:17:23 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 18:17:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:17:28 Rebalance progress: 12.5
2021/12/04 18:17:33 Rebalance progress: 12.5
2021/12/04 18:17:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:17:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:17:43 Rebalance progress: 12.5
2021/12/04 18:17:48 Rebalance progress: 12.5
2021/12/04 18:17:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:17:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:17:53 Waiting for src bucket item count to get to: 50000 curr count: 47540
2021/12/04 18:17:58 Waiting for src bucket item count to get to: 50000 curr count: 47540
2021/12/04 18:18:08 Waiting for dst bucket item count to get to: 50000 curr count: 42251
2021/12/04 18:18:13 Waiting for dst bucket item count to get to: 50000 curr count: 42251
2021/12/04 18:18:18 Waiting for dst bucket item count to get to: 50000 curr count: 48232
2021/12/04 18:18:23 Waiting for dst bucket item count to get to: 50000 curr count: 48232
2021/12/04 18:18:23 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:18:23 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:18:24 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:18:29 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:34 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:39 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:44 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:49 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:54 Waiting for app: TestDuplicateTimerFiringEventingSwapReb to get un-deployed
2021/12/04 18:18:54 App: TestDuplicateTimerFiringEventingSwapReb got un-deployed
2021/12/04 18:18:54 No eventing-con process running
2021/12/04 18:18:54 Function: TestDuplicateTimerFiringEventingSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringEventingSwapReb deleting in the background"
}
2021/12/04 18:18:56 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:19:01 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:19:06 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:19:06 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:19:08 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:19:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:19:18 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:19:18 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringEventingSwapReb (180.22s)
=== RUN   TestDuplicateTimerFiringKVRebIn
2021/12/04 18:19:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:19:28 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 18:19:28 No eventing-con process running
2021/12/04 18:19:28 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebIn not found"
 }
}
2021/12/04 18:19:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:19:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:19:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:19:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:19:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:19:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:19:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:19:46 Deploying app: TestDuplicateTimerFiringKVRebIn. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:19:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebIn' in metakv",
  "warnings": null
 }
}
2021/12/04 18:19:52 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 18:19:57 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 18:20:02 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 18:20:07 Waiting for app: TestDuplicateTimerFiringKVRebIn to get deployed
2021/12/04 18:20:07 App: TestDuplicateTimerFiringKVRebIn got deployed
2021/12/04 18:20:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:20:07 Starting bucket ops to source bucket
2021/12/04 18:20:08 Removing node(s): [] from the cluster
2021/12/04 18:20:13 Rebalance progress: 12.5
2021/12/04 18:20:18 Rebalance progress: 12.5
2021/12/04 18:20:23 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:20:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:20:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:20:28 Rebalance progress: 12.5
2021/12/04 18:20:33 Rebalance progress: 12.5
2021/12/04 18:20:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:20:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:20:38 Waiting for src bucket item count to get to: 50000 curr count: 27335
2021/12/04 18:20:43 Waiting for src bucket item count to get to: 50000 curr count: 36859
2021/12/04 18:20:48 Waiting for src bucket item count to get to: 50000 curr count: 36859
2021/12/04 18:20:53 Waiting for src bucket item count to get to: 50000 curr count: 49055
2021/12/04 18:20:58 Waiting for src bucket item count to get to: 50000 curr count: 49055
2021/12/04 18:21:08 Waiting for dst bucket item count to get to: 50000 curr count: 42455
2021/12/04 18:21:13 Waiting for dst bucket item count to get to: 50000 curr count: 42455
2021/12/04 18:21:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:21:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:21:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:21:18 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:23 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:28 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:34 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:39 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:44 Waiting for app: TestDuplicateTimerFiringKVRebIn to get un-deployed
2021/12/04 18:21:44 App: TestDuplicateTimerFiringKVRebIn got un-deployed
2021/12/04 18:21:44 No eventing-con process running
2021/12/04 18:21:44 Function: TestDuplicateTimerFiringKVRebIn delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebIn deleting in the background"
}
2021/12/04 18:21:45 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:21:50 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:21:55 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:21:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:21:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:22:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:22:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:22:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebIn (169.69s)
=== RUN   TestDuplicateTimerFiringKVRebOut
2021/12/04 18:22:12 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:22:13 Removing node(s): [] from the cluster
2021/12/04 18:22:18 Rebalance progress: 100
2021/12/04 18:22:23 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:22:23 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 18:22:23 No eventing-con process running
2021/12/04 18:22:23 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVRebOut not found"
 }
}
2021/12/04 18:22:25 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:22:30 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:22:30 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:22:32 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:22:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:22:37 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:22:42 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:22:42 Deploying app: TestDuplicateTimerFiringKVRebOut. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:22:43 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVRebOut' in metakv",
  "warnings": null
 }
}
2021/12/04 18:22:48 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 18:22:53 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 18:22:58 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 18:23:03 Waiting for app: TestDuplicateTimerFiringKVRebOut to get deployed
2021/12/04 18:23:03 App: TestDuplicateTimerFiringKVRebOut got deployed
2021/12/04 18:23:03 Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:23:03 Starting bucket ops to source bucket
2021/12/04 18:23:08 Rebalance progress: 12.5
2021/12/04 18:23:13 Rebalance progress: 12.5
2021/12/04 18:23:18 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:23:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:23:18 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:23:18 Waiting for src bucket item count to get to: 50000 curr count: 8459
2021/12/04 18:23:23 Waiting for src bucket item count to get to: 50000 curr count: 21923
2021/12/04 18:23:28 Waiting for src bucket item count to get to: 50000 curr count: 21923
2021/12/04 18:23:33 Waiting for src bucket item count to get to: 50000 curr count: 33341
2021/12/04 18:23:38 Waiting for src bucket item count to get to: 50000 curr count: 33341
2021/12/04 18:23:43 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/04 18:23:48 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/04 18:23:53 Waiting for src bucket item count to get to: 50000 curr count: 43353
2021/12/04 18:24:03 Waiting for dst bucket item count to get to: 50000 curr count: 34333
2021/12/04 18:24:08 Waiting for dst bucket item count to get to: 50000 curr count: 34333
2021/12/04 18:24:13 Waiting for dst bucket item count to get to: 50000 curr count: 48208
2021/12/04 18:24:18 Waiting for dst bucket item count to get to: 50000 curr count: 48208
2021/12/04 18:24:18 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:24:18 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:24:18 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:24:23 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:28 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:33 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:38 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:43 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:48 Waiting for app: TestDuplicateTimerFiringKVRebOut to get un-deployed
2021/12/04 18:24:48 App: TestDuplicateTimerFiringKVRebOut got un-deployed
2021/12/04 18:24:48 No eventing-con process running
2021/12/04 18:24:48 Function: TestDuplicateTimerFiringKVRebOut delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVRebOut deleting in the background"
}
2021/12/04 18:24:50 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:24:55 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:25:00 Waiting for dst bucket item count to get to: 0 curr count: 13725
2021/12/04 18:25:05 Waiting for dst bucket item count to get to: 0 curr count: 13725
2021/12/04 18:25:05 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:25:07 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:25:12 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:25:17 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:25:17 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVRebOut (189.83s)
=== RUN   TestDuplicateTimerFiringKVSwapReb
2021/12/04 18:25:27 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:25:27 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 18:25:27 No eventing-con process running
2021/12/04 18:25:27 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: TestDuplicateTimerFiringKVSwapReb not found"
 }
}
2021/12/04 18:25:29 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:25:34 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:25:34 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:25:36 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:25:41 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:25:41 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:25:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:25:46 Deploying app: TestDuplicateTimerFiringKVSwapReb. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:25:47 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'TestDuplicateTimerFiringKVSwapReb' in metakv",
  "warnings": null
 }
}
2021/12/04 18:25:52 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 18:25:57 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 18:26:02 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 18:26:07 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get deployed
2021/12/04 18:26:07 App: TestDuplicateTimerFiringKVSwapReb got deployed
2021/12/04 18:26:07 Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:26:07 Starting bucket ops to source bucket
2021/12/04 18:26:07 Removing node(s): [] from the cluster
2021/12/04 18:26:12 Rebalance progress: 12.5
2021/12/04 18:26:17 Rebalance progress: 12.5
2021/12/04 18:26:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:26:22 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:26:23 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:26:28 Rebalance progress: 12.5
2021/12/04 18:26:33 Rebalance progress: 12.5
2021/12/04 18:26:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:26:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:26:43 Rebalance progress: 12.5
2021/12/04 18:26:48 Rebalance progress: 12.5
2021/12/04 18:26:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:26:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:26:53 Waiting for src bucket item count to get to: 50000 curr count: 46918
2021/12/04 18:26:58 Waiting for src bucket item count to get to: 50000 curr count: 46918
2021/12/04 18:27:08 Waiting for dst bucket item count to get to: 50000 curr count: 46110
2021/12/04 18:27:13 Waiting for dst bucket item count to get to: 50000 curr count: 46110
2021/12/04 18:27:13 src & dst bucket item count matched up. src bucket count: 50000 dst bucket count: 50000
2021/12/04 18:27:13 Post SET ops. Expected item count: 50000 got 50000
2021/12/04 18:27:13 Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:27:18 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:27:23 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:27:28 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:27:33 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:27:39 Waiting for app: TestDuplicateTimerFiringKVSwapReb to get un-deployed
2021/12/04 18:27:39 App: TestDuplicateTimerFiringKVSwapReb got un-deployed
2021/12/04 18:27:39 No eventing-con process running
2021/12/04 18:27:39 Function: TestDuplicateTimerFiringKVSwapReb delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: TestDuplicateTimerFiringKVSwapReb deleting in the background"
}
2021/12/04 18:27:40 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:27:45 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:27:50 Waiting for dst bucket item count to get to: 0 curr count: 23014
2021/12/04 18:27:55 Waiting for dst bucket item count to get to: 0 curr count: 23014
2021/12/04 18:27:55 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:27:57 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:28:07 Waiting for dst bucket item count to get to: 0 curr count: 50000
2021/12/04 18:28:07 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestDuplicateTimerFiringKVSwapReb (170.17s)
=== RUN   TestKVRebOnUpdateBucketOpOneByOne
2021/12/04 18:28:17 Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:28:17 App: bucket_op_on_update got un-deployed
2021/12/04 18:28:17 No eventing-con process running
2021/12/04 18:28:17 Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:28:19 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:24 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:28:24 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:28:26 Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:31 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:28:31 src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:28:36 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:28:37 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:28:42 Starting bucket ops to source bucket
2021/12/04 18:28:47 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:28:52 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:28:57 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:28:57 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:28:57 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:28:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:28:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:29:03 Rebalance progress: 12.5
2021/12/04 18:29:08 Rebalance progress: 12.5
2021/12/04 18:29:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:29:13 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:29:13 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:29:18 Rebalance progress: 12.5
2021/12/04 18:29:23 Rebalance progress: 12.5
2021/12/04 18:29:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:29:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 18:29:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:29:33 Rebalance progress: 12.5
2021/12/04 18:29:38 Rebalance progress: 12.5
2021/12/04 18:29:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:29:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:29:49 Rebalance progress: 12.5
2021/12/04 18:29:54 Rebalance progress: 12.5
2021/12/04 18:29:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:29:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:29:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:30:04 Rebalance progress: 12.5
2021/12/04 18:30:09 Rebalance progress: 12.5
2021/12/04 18:30:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:30:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:30:14 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 18:30:19 Rebalance progress: 12.5
2021/12/04 18:30:24 Rebalance progress: 12.5
2021/12/04 18:30:29 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:30:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:30:29 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:30:29 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:30:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:30:59 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:30:59 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:30:59 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:31:01 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:31:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:31:11 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9520
2021/12/04 18:31:16 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9520
2021/12/04 18:31:16 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:31:18 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:31:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:31:28 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 10670
2021/12/04 18:31:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpOneByOne (200.50s)
=== RUN   TestKVForceFailover
2021/12/04 18:31:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:31:43 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:31:43 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:31:43 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:31:45 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:31:50 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:31:50 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:31:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:31:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:31:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:31:56 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:31:57 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:32:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:32:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:32:02 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:32:03 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:32:03 Starting bucket ops to source bucket
2021/12/04 18:32:08 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:32:13 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:32:18 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:32:23 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:32:23 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:32:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:32:23 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:32:23 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 18:32:43 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 18:32:43 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: eventing to the cluster
2021/12/04 18:32:49 Rebalance progress: 12.5
2021/12/04 18:32:54 Rebalance progress: 12.5
2021/12/04 18:32:59 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:32:59 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:32:59 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002] from the cluster
2021/12/04 18:33:04 Rebalance progress: 12.5
2021/12/04 18:33:09 Rebalance progress: 12.5
2021/12/04 18:33:14 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:33:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:33:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:33:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:33:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:33:44 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:33:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:33:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:33:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:33:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7051
2021/12/04 18:33:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:33:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:34:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7051
2021/12/04 18:34:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVForceFailover (160.26s)
=== RUN   TestKVHardFailoverRecovery
2021/12/04 18:34:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:34:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:34:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:34:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:34:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:34:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:34:27 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:32 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:34:32 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:37 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:34:37 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:34:38 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:34:43 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:34:48 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:34:53 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:34:58 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:34:58 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:34:58 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:34:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:34:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:35:03 Rebalance progress: 12.5
2021/12/04 18:35:08 Rebalance progress: 12.5
2021/12/04 18:35:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:35:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:35:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
2021/12/04 18:36:13 clusterSetup.go:237: Kicking off failover recovery, type: full
2021/12/04 18:36:18 Rebalance progress: 12.5
2021/12/04 18:36:23 Rebalance progress: 12.5
2021/12/04 18:36:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:36:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:36:28 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:36:33 Rebalance progress: 12.5
2021/12/04 18:36:38 Rebalance progress: 12.5
2021/12/04 18:36:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:36:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:36:44 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:36:49 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:36:54 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:36:59 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:37:04 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:37:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:37:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:37:14 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:37:14 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:37:14 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:37:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:37:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:37:26 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12062
2021/12/04 18:37:26 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:37:28 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:37:33 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:37:38 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12062
2021/12/04 18:37:38 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVHardFailoverRecovery (209.52s)
=== RUN   TestKVRebOnUpdateBucketOpAllAtOnce
2021/12/04 18:37:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:37:48 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:37:48 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:37:48 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:37:49 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:37:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:37:54 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:37:56 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:01 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:38:01 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:06 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:38:06 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:38:07 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:38:12 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:38:17 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:38:22 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:38:27 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:38:27 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:27 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:38:27 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:38:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:38:28 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 18:38:28 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:38:33 Rebalance progress: 12.5
2021/12/04 18:38:38 Rebalance progress: 12.5
2021/12/04 18:38:43 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:43 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:38:43 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001 127.0.0.1:19002 127.0.0.1:19003] from the cluster
2021/12/04 18:38:48 Rebalance progress: 12.5
2021/12/04 18:38:53 Rebalance progress: 12.5
2021/12/04 18:38:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:38:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:38:58 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:39:03 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:09 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:39:29 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:39:29 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:39:29 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:39:31 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:39:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:39:41 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2571
2021/12/04 18:39:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2571
2021/12/04 18:39:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:39:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:39:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:39:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 4594
2021/12/04 18:39:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebOnUpdateBucketOpAllAtOnce (139.81s)
=== RUN   TestKVSwapRebOnUpdateBucketOp
2021/12/04 18:40:07 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:40:07 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:40:07 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:40:07 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:40:09 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:14 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:40:14 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:40:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:40:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:40:21 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:40:22 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:40:27 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:40:32 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:40:37 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:40:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:40:42 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:42 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:40:42 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:40:42 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:40:48 Rebalance progress: 12.5
2021/12/04 18:40:53 Rebalance progress: 12.5
2021/12/04 18:40:58 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:40:58 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:40:58 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:40:58 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:41:03 Rebalance progress: 12.5
2021/12/04 18:41:08 Rebalance progress: 12.5
2021/12/04 18:41:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:41:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:41:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:41:13 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:41:18 Rebalance progress: 12.5
2021/12/04 18:41:23 Rebalance progress: 12.5
2021/12/04 18:41:28 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:41:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:41:28 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:41:28 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:41:33 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:38 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:43 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:48 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:53 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:58 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:41:58 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:41:58 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:41:58 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:42:00 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:42:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:42:10 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2092
2021/12/04 18:42:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 2092
2021/12/04 18:42:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:42:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:42:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:42:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6081
2021/12/04 18:42:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:42:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:42:32 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:42:32 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:42:32 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:42:34 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:42:39 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:42:39 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:42:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:42:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:42:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebOnUpdateBucketOp (168.42s)
=== RUN   TestKVRebalanceWithMultipleHandlers
2021/12/04 18:42:56 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:42:56 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:42:56 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:42:56 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:42:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:03 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:43:04 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:09 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:09 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:43:14 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:43:14 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 18:43:14 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:43:14 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 18:43:16 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:21 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:21 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:43:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:28 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:43:29 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:43:29 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:43:29 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:43:30 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 18:43:35 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:43:40 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:43:45 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:43:50 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:43:50 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 18:43:55 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:44:00 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:44:05 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:44:05 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:05 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:44:05 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:44:06 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:44:11 Rebalance progress: 12.5
2021/12/04 18:44:16 Rebalance progress: 12.5
2021/12/04 18:44:21 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:21 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:44:21 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:44:21 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:44:26 Rebalance progress: 12.5
2021/12/04 18:44:31 Rebalance progress: 12.5
2021/12/04 18:44:36 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:36 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:44:36 clusterSetup.go:245: Adding node: https://127.0.0.1:19003 with role: kv to the cluster
2021/12/04 18:44:36 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:44:41 Rebalance progress: 12.5
2021/12/04 18:44:46 Rebalance progress: 12.5
2021/12/04 18:44:51 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:44:51 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:44:51 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:44:56 Rebalance progress: 12.5
2021/12/04 18:45:01 Rebalance progress: 12.5
2021/12/04 18:45:06 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:06 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:45:06 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:45:12 Rebalance progress: 12.5
2021/12/04 18:45:17 Rebalance progress: 12.5
2021/12/04 18:45:22 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:22 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:45:22 clusterSetup.go:253: Removing node(s): [127.0.0.1:19003] from the cluster
2021/12/04 18:45:27 Rebalance progress: 12.5
2021/12/04 18:45:32 Rebalance progress: 12.5
2021/12/04 18:45:37 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:45:37 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:45:37 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:45:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:45:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:45:52 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:45:57 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:46:02 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:46:02 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:46:02 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:46:02 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:46:05 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:46:10 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:46:15 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12115
2021/12/04 18:46:15 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:46:17 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:46:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:46:27 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 12115
2021/12/04 18:46:27 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:46:27 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:46:32 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:37 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:42 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:47 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:52 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:57 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:46:57 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 18:46:57 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:46:57 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 18:46:59 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:04 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:04 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:47:06 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:11 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:11 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebalanceWithMultipleHandlers (265.30s)
=== RUN   TestKVSwapRebalanceWithMultipleHandlers
2021/12/04 18:47:21 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:47:21 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:47:21 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:47:21 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:47:23 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:28 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:28 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:47:30 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:35 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:35 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:47:40 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:47:40 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 18:47:40 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:47:40 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: sys_test_bucket_op not found"
 }
}
2021/12/04 18:47:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:47:48 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:53 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:53 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:53 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:47:54 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:47:54 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:47:55 eventingSetup.go:125: Deploying app: sys_test_bucket_op. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:47:56 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'sys_test_bucket_op' in metakv",
  "warnings": null
 }
}
2021/12/04 18:48:01 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:48:06 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:48:11 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:48:16 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:48:16 clusterSetup.go:354: App: bucket_op_on_update got deployed
2021/12/04 18:48:21 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:48:26 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:48:31 clusterSetup.go:346: Waiting for app: sys_test_bucket_op to get deployed
2021/12/04 18:48:31 clusterSetup.go:354: App: sys_test_bucket_op got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:48:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:48:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:48:31 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:48:31 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:48:36 Rebalance progress: 12.5
2021/12/04 18:48:41 Rebalance progress: 12.5
2021/12/04 18:48:46 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:48:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:48:46 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:48:46 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:48:47 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:48:52 Rebalance progress: 12.5
2021/12/04 18:48:57 Rebalance progress: 12.5
2021/12/04 18:49:02 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:49:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:49:02 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:49:02 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:49:07 Rebalance progress: 12.5
2021/12/04 18:49:12 Rebalance progress: 12.5
2021/12/04 18:49:17 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:49:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:49:17 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:49:17 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:49:22 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:27 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:32 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:37 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:42 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:47 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:49:47 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:49:47 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:49:47 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:49:51 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:49:56 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:50:01 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1822
2021/12/04 18:50:06 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 1822
2021/12/04 18:50:06 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:50:08 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:50:13 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:50:18 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 7606
2021/12/04 18:50:18 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:50:18 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:50:23 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:28 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:33 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:38 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:43 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:48 clusterSetup.go:397: Waiting for app: sys_test_bucket_op to get un-deployed
2021/12/04 18:50:48 clusterSetup.go:405: App: sys_test_bucket_op got un-deployed
2021/12/04 18:50:48 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:50:48 eventingSetup.go:418: Function: sys_test_bucket_op delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: sys_test_bucket_op deleting in the background"
}
2021/12/04 18:50:50 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:50:55 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:50:55 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:50:57 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:02 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:51:02 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVSwapRebalanceWithMultipleHandlers (230.77s)
=== RUN   TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne
2021/12/04 18:51:12 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:51:12 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:51:12 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:51:12 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:51:14 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:19 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:51:19 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:51:20 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:25 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:51:25 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:30 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:51:30 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:51:32 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:51:37 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:51:42 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:51:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:51:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:51:52 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:51:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:51:52 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:51:52 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:52:12 clusterSetup.go:165: Stopping rebalance
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:12 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:22 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:52:22 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:52:28 Rebalance progress: 12.5
2021/12/04 18:52:33 Rebalance progress: 12.5
2021/12/04 18:52:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:52:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:52:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:52:43 Rebalance progress: 12.5
2021/12/04 18:52:48 Rebalance progress: 12.5
2021/12/04 18:52:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:52:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:52:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:52:58 Rebalance progress: 12.5
2021/12/04 18:53:03 Rebalance progress: 12.5
2021/12/04 18:53:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:53:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:53:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:53:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:53:39 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:53:39 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:53:39 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:53:41 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:53:46 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:53:51 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6301
2021/12/04 18:53:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 6301
2021/12/04 18:53:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:53:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:54:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 9176
2021/12/04 18:54:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVRebStopStartKVOpsOnUpdateBucketOpOneByOne (185.70s)
=== RUN   TestKVFailoverOnUpdateBucketOp
2021/12/04 18:54:18 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:54:18 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:54:18 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:54:18 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 404 dump: {
 "name": "ERR_APP_NOT_FOUND_TS",
 "code": 13,
 "description": "Function not found in temporary store",
 "attributes": null,
 "runtime_info": {
  "code": 13,
  "info": "Function: bucket_op_on_update not found"
 }
}
2021/12/04 18:54:19 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:24 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:54:24 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:54:26 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:31 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:54:31 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:36 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:54:36 eventingSetup.go:125: Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:54:37 eventingSetup.go:102: Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:54:42 bucketOps.go:157: Starting bucket ops to source bucket
2021/12/04 18:54:47 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:54:52 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:54:57 clusterSetup.go:346: Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:54:57 clusterSetup.go:354: App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:54:57 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:54:57 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:54:58 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:55:03 Rebalance progress: 12.5
2021/12/04 18:55:08 Rebalance progress: 12.5
2021/12/04 18:55:13 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:13 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:55:13 clusterSetup.go:229: Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:23 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:55:23 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:55:28 Rebalance progress: 12.5
2021/12/04 18:55:33 Rebalance progress: 12.5
2021/12/04 18:55:38 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:55:38 clusterSetup.go:245: Adding node: https://127.0.0.1:19002 with role: kv to the cluster
2021/12/04 18:55:38 clusterSetup.go:253: Removing node(s): [127.0.0.1:19001] from the cluster
2021/12/04 18:55:43 Rebalance progress: 12.5
2021/12/04 18:55:48 Rebalance progress: 12.5
2021/12/04 18:55:53 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:55:53 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:55:53 clusterSetup.go:253: Removing node(s): [127.0.0.1:19002] from the cluster
2021/12/04 18:55:58 Rebalance progress: 12.5
2021/12/04 18:56:03 Rebalance progress: 12.5
2021/12/04 18:56:08 Rebalance progress: 100
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:56:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:56:08 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:56:09 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:56:14 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:56:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:56:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:56:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:56:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:56:34 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:56:34 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:56:34 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:56:36 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:56:41 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:56:46 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8610
2021/12/04 18:56:46 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:56:47 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:56:52 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:56:57 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 8610
2021/12/04 18:56:57 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestKVFailoverOnUpdateBucketOp (169.96s)
=== RUN   TestBootstrapAfterKVHardFailover
2021/12/04 18:57:02 clusterSetup.go:245: Adding node: https://127.0.0.1:19001 with role: kv to the cluster
2021/12/04 18:57:03 clusterSetup.go:253: Removing node(s): [] from the cluster
2021/12/04 18:57:08 Rebalance progress: 100
2021/12/04 18:57:08 Failing over: [127.0.0.1:19001]
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:57:38 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:57:38 Deploying app: bucket_op_on_update. Item count src bucket: 0 dst bucket: 0
2021/12/04 18:57:39 Context: Post to main store, response code: 200 dump: {
 "code": 0,
 "info": {
  "status": "Stored function: 'bucket_op_on_update' in metakv",
  "warnings": null
 }
}
2021/12/04 18:57:44 Removing node(s): [] from the cluster
2021/12/04 18:57:44 Starting bucket ops to source bucket
2021/12/04 18:57:49 Rebalance failed. See logs for detailed reason. You can try again.
2021/12/04 18:57:54 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:57:59 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:58:04 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:58:09 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:58:14 Waiting for app: bucket_op_on_update to get deployed
2021/12/04 18:58:14 App: bucket_op_on_update got deployed
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

VB distribution across eventing nodes and their workers::

Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:58:14 waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 

DCP Stream statuses:

vbucket curr owner:


2021/12/04 18:58:14 eventingSetup.go:586: Unmarshal undeploy response: unexpected end of JSON input
2021/12/04 18:58:19 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:24 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:29 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:34 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:39 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:44 clusterSetup.go:397: Waiting for app: bucket_op_on_update to get un-deployed
2021/12/04 18:58:44 clusterSetup.go:405: App: bucket_op_on_update got un-deployed
2021/12/04 18:58:44 eventingSetup.go:46: No eventing-con process running
2021/12/04 18:58:44 eventingSetup.go:418: Function: bucket_op_on_update delete from main store request response code: 200 dump: {
 "code": 0,
 "info": "Function: bucket_op_on_update deleting in the background"
}
2021/12/04 18:58:46 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:58:51 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:58:56 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/04 18:58:56 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
2021/12/04 18:58:58 eventingSetup.go:102: Context: Bucket flush, response code: 200 dump: 
Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused
2021/12/04 18:59:03 clusterSetup.go:81: waitForIndexes giving up: err Get "http://127.0.0.1:9108/getIndexStatus": dial tcp 127.0.0.1:9108: connect: connection refused response 
2021/12/04 18:59:08 eventingSetup.go:441: Waiting for dst bucket item count to get to: 0 curr count: 3000
2021/12/04 18:59:08 eventingSetup.go:436: src & dst bucket item count matched up. src bucket count: 0 dst bucket count: 0
--- PASS: TestBootstrapAfterKVHardFailover (130.67s)
PASS
ok  	github.com/couchbase/eventing/tests/functional_tests	12167.962s

Rebalance tests took 3 hours, 22 minutes and 50 seconds.

Testing Succeeded


Version: versions-04.12.2021-11.15.cfg
Build Log: make-04.12.2021-11.15.log
Server Log: logs-04.12.2021-11.15.tar.gz

Finished